AFRL-HE-WP-TR-2004-01 66 


STINFO  COPY 
United  States  Air  Force 
Research  Laboratory 


Logistics  Control  and  Information 
Support  (LOCIS) 


Gary  Hardenburg 
Clark  Moskop 
John  Potts 


BAE  Systems  Mission  Solutions 
P.O. 509009 
San  Diego,  CA  92150 


Christopher  K.  Curtis 
Kelly  Vinson 
David  Lemery 

Air  Force  Research  Laboratory 


September  2004 


Final  Report  for  the  Period  September  1999  to  September  2004 


20050425  064 


_ ’  Human  Effectiveness  Directorate 

”  Warfighter  Readiness  Research  Division 

Approved  for  public  release;  distribution  is  unlimited.  Logistics  Readiness  Branch 

_ _ _ 1  2698  G  Street 

Wright-Patterson  AFB  OH  45433-7604 


NOTICES 


When  US  Government  drawings,  specifications  or  other  data  are  used  for  any  purpose  other  than  a 
definitely  related  Government  procurement  operation,  the  Government  thereby  incurs  no  responsibility  nor 
any  obligation  whatsoever,  and  the  fact  that  the  Government  may  have  formulated,  furnished,  or  in  any  way 
supplied  the  said  drawings,  specifications  or  other  data,  is  not  to  be  regarded  by  implication  or  otherwise,  as 
in  any  manner  licensing  the  holder  or  any  other  person  or  corporation,  or  conveying  any  rights  or 
permission  to  manufacture,  use,  or  sell  any  patented  invention  that  may  in  any  way  be  related  thereto. 

Please  do  not  request  copies  of  this  report  from  the  Air  Force  Research  Laboratory.  Additional  copies  may 
be  purchased  from: 


National  Technical  Information  Service 
5285  Port  Royal  Road 
Springfield,  V A  22161 

Federal  Government  agencies  and  their  contractors  registered  with  the  Defense  Technical  Information 
Center  should  direct  requests  for  copies  of  this  report  to: 

Defense  Technical  Information  Center 
8725  John  J.  Kingman  Rd.,  Ste  0944 
Ft.  Belvoir,  VA  22060-6218 


TECHNICAL  REVIEW  AND  APPROVAL 
AFRL-HE- WP-TR-2004-0 1 66 


This  report  has  been  reviewed  by  the  Office  of  Public  Affairs  (PA)  and  is  releasable  to  the  National 
Technical  Information  Service  (NTIS).  At  NTIS,  it  will  be  available  to  the  general  public,  including 
foreign  nations. 


This  technical  report  has  been  reviewed  and  is  approved  for  publication. 


FOR  THE  COMMANDER 
//SIGNED// 

CURTIS  J.  PAPKE,  Colonel,  USAF 
Chief  Warfighter  Readiness  Research  Division 
Air  Force  Research  Laboratory 


REPORT  DOCUMENTATION  PAGE 


Form  Approved 
OMB  No.  0704-0188 


Public  reporting  burden  few*  this  collection  of  information  is  estimated  to  average  1  hour  per  response,  including  the  time  for  reviewing  instructions,  searching  existing  data  sources,  gathering  and  maintaining  the 
data  needed,  and  completing  and  reviewing  this  collection  of  information.  Send  comments  regarding  this  burden  estimate  or  any  other  aspect  of  this  collection  of  information,  including  suggestions  for  reducing 
this  burden  to  Department  of  Defense,  Washington  Headquarters  Services,  Directorate  for  Information  Operations  and  Reports  (0704-0188),  1215  Jefferson  Davis  Highway,  Suite  1204,  Arlington,  VA  22202- 
4302.  Respondents  should  be  aware  that  notwithstanding  any  other  provision  of  law,  no  person  shall  be  subject  to  any  penalty  for  failing  to  comply  with  a  collection  of  information  if  it  does  not  display  a  currently 
valid  OMB  control  number.  PLEASE  DO  NOT  RETURN  YOUR  FORM  TO  THE  ABOVE  ADDRESS. _ 


1.  REPORT  DATE  (DD-MM-YYYY) 


2.  REPORT  TYPE 


September  2004 


4.  TITLE  AND  SUBTITLE 


Final 


3.  DATES  COVERED  (From  -  To) 
September  1999  -  September  2004 

5a.  CONTRACT  NUMBER 


Logistics  Control  and  Information  Support  (LOCIS) 


F33615-99-C-6010 

5b.  GRANT  NUMBER 


6.  AUTHOR (S) 

Gary  Hardenburg,  Clark  Moskop,  John  Potts,  Christopher  K. 
Curtis,  Kelly  Vinson,  David  Lemery 


7.  PERFORMING  ORGANIZATION  NAME(S)  AND  ADDRESS(ES) 


5c.  PROGRAM  ELEMENT  NUMBER 

63231F 

5d.  PROJECT  NUMBER 

4923 

5e.  TASK  NUMBER 

02 

5f.  WORK  UNIT  NUMBER 

03 

8.  PERFORMING  ORGANIZATION  REPORT 
NUMBER 


BAE  Systems  Mission  Solutions 
P.O.  509009 


San  Diego,  CA  92150 


1007832 


9.  SPONSORING  /  MONITORING  AGENCY  NAME(S)  AND  ADDRESS(ES) 


10.  SPONSOR/MONITOR’S  ACRONYM(S) 


Air  Force  Research  Laboratory 


Human  Effectiveness  Directorate 

Warfighter  Readiness  Research  Division 

Logistics  Readiness  Branch 

Wright -Patterson  AFB  OH  45433-7604 _ 


11.  SPONSOR/MONITOR’S  REPORT 
NUMBER(S) 

AFRL-HE-WP-TR-2004- 0166 


12.  DISTRIBUTION  /  AVAILABILITY  STATEMENT 


Approved  for  public  release;  distribution  is  unlimited. 

13.  SUPPLEMENTARY  NOTES 


14.  ABSTRACT 

This  technical  report  contains  the  results  of  the  Air  Force  Logistics  Control  and  Information  Support 
(LOCIS)  project.  The  LOCIS  objective  was  to  perform  logistics  research  and  development  in  a  series  of 
spirals  that  concentrated  on  information  technologies  applied  to  wing  level  decision  makers  and  to 
support  the  Agile  Combat  Support  and  the  Air  Expeditionary  Force.  More  specifically,  the  objective  was 
to  explore,  develop  and  integrate  information  technologies  that  automatically  access  critical 
information;  display  clear  and  concise  information;  help  the  user  solve  difficult  problems  by 
suggesting  alternative  courses  of  action;  and  autonomously  take  action  that  benefits  the  user.  These 
objectives  were  implemented  through  the  following  technological  areas:  Information  Fusion,  Proactive 
Decision  Support,  and  Dynamic  User  Interface.  In  summary,  an  effective  LOCIS  system  will  provide 
multiple  benefits  including  more  timely  information,  minimized  search  time,  reduced  data 
interpretation,  predictive  simulations,  and  dynamic  re-planning. 


15.  SUBJECT  TERMS  Logistics  Control  and  Information  Support  (LOCIS),  Advanced  Decision  Support  System, 
Command  and  Control  Systems,  Decision  Making,  Decision  Support  Systems,  Information  Systems,  Logistics 


16.  SECURITY  CLASSIFICATION  OF: 

17.  LIMITATION 

OF  ABSTRACT 

18.  NUMBER 
OF  PAGES 

19a.  NAME  OF  RESPONSIBLE  PERSON 

a.  REPORT 

UNCLASSIFIED 

b.  ABSTRACT 

UNCLASSIFIED 

c.  THIS  PAGE 

UNCLASSIFIED 

SAR 

120 

19b.  TELEPHONE  NUMBER  ( include  area 
code) 

1 


Standard  Form  298  (Rev.  8-98) 

Prescribed  by  ANSI  Std.  239.18 


THIS  PAGE  LEFT  INTENTIONALLY  BLANK 


11 


Preface 


This  technical  report  contains  the  results  of  the  Air  Force  Logistics  Control  and  Information 
Support  (LOCIS)  project.  This  work  was  executed  under  the  Air  Force  Research  Laboratories 
contract  F33615-99-C-6010.  The  work  described  in  this  report  was  performed  from 
15  September  1999  through  30  September  2004.  The  objective  of  this  task  was  to  perform 
logistics  research  and  development  in  a  series  of  spirals  that  concentrated  on  information 
technologies  applied  to  wing  level  decision  makers  and  to  support  the  Agile  Combat  Support  and 
the  Air  Expeditionary  Force.  More  specifically,  the  objective  was  to  explore,  develop  and 
integrate  information  technologies  that  automatically  access  critical  information;  display  clear 
and  concise  information;  help  the  user  solve  difficult  problems  by  suggesting  alternative  courses 
of  action;  and  autonomously  take  action  that  benefits  the  user.  These  objectives  were 
implemented  through  the  following  technological  areas:  Information  Fusion,  Proactive  Decision 
Support,  and  Dynamic  User  Interface.  Also  included  in  this  report  is  a  paper  describing  the  initial 
effort  to  measure  Situation  Awareness  (SA)  in  the  context  of  logistics  command  and  control  (see 
Appendix  E).  LOCIS  is  a  prototype  decision  aid  intended  to  allow  a  maintenance  supervisor  to 
perform  his/her  job  more  quickly  without  any  loss  of  SA. 

The  principal  investigators  for  this  effort  included  Mr.  Chris  Curtis,  Capt.  Kelly  Vinson  and 
Lt.  David  Lemery  from  AFRL,  Mr.  Gary  Hardenburg,  Mr.  Clark  Moskop  and  Mr.  John  Potts 
from  BAE  Systems,  Mr.  Rick  Iannacchione  and  Ms.  Deb  Fannon  from  Kelley’s  Logistics 
Support  Systems,  Dr.  Deborah  Mitta  from  Georgia  Tech  Research  Institute  and  Ms.  LaDonna 
Schneller  and  Mr.  Rob  Roy  from  Decision  Sciences  Inc. 
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Overview 


This  Final  Report  documents  the  results  of  the  Logistics  Control  and  Information  Support 
(LOCIS)  Research  and  Development  program.  LOCIS  was  a  42  month  project,  sponsored  by  the 
Air  Force  Research  Laboratory  at  Wright  Patterson  Air  Force  Base  (AFB),  supported  by  the  Air 
Force  Command  and  Control  Intelligence  Surveillance  Reconnaissance  Center  (AFC2ISRC)  at 
Langley  AFB.  Other  stakeholders  included  ACC/LGX,  AFSOC/LG,  and  SSG/IL.  BAE  Systems 
was  the  prime  contractor  with  key  support  provided  by  Kelley’s  Logistics  Support  System 
(KLSS),  Georgia  Tech  Research  Institute  (GTRJ)  and  Decision  Sciences  Inc.  (DSI). 

i 

The  program  began  in  September  1999  and  included  demonstrations  each  year  through  Spiral  3. 
Figure  1  shows  the  spiral  model  of  the  42-month  program,  which  began  the  first  year  focused  on 
the  Operations  Group  Commander  (OG),  the  Logistics  Group  Commander  (LG)  and  their 
associated  staff.  Subsequent  years  included  lower  levels  of  command,  such  as  the  production 
supervisor  and  other  areas  of  concern,  such  as  munitions.  Each  spiral  also  added  additional 
capabilities,  such  as  in  Spiral  3  when  deployment  planning  and  munitions  tracking  were 
incorporated.  The  results  of  each  yearly  spiral  fed  into  the  next  year’s  research  through  lessons 
learned  and  comments  from  users  and  those  present  at  the  many  demonstrations. 
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Additional  User  Profiles 
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Figure  1.  LOCIS  Spiral  Model 


Each  spiral  also  contained  a  series  of  quantitative  and  qualitative  reports  that  assessed  the  success 
of  the  specific  capabilities  demonstrated  that  year  and  performed  a  cost  benefit  analysis  of  the 
specific  capabilities.  This  report  contains  a  summary  of  each  spiral  and  the  overall  results  of  the 
quantitative  and  qualitative  analysis  with  specific  emphasis  on  the  final  spiral. 

2  Objective 

The  LOCIS  program  objective  was  to  perform  logistics  research  and  development  in  a  series  of 
spirals  that  concentrated  on  information  technologies  applied  to  wing  level  decision  makers  and 
to  support  the  Agile  Combat  Support  and  the  Air  Expeditionary  Force.  More  specifically,  the 
objective  was  to  explore,  develop  and  integrate  information  technologies  that  automatically 
access  critical  information;  display  clear  and  concise  information;  help  the  user  solve  difficult 
problems  by  suggesting  alternative  courses  of  action;  and  autonomously  take  action  that  benefits 
the  user.  These  objectives  were  implemented  through  the  following  technological  areas: 
Information  Fusion,  Proactive  Decision  Support,  and  Dynamic  User  Interface. 

3  Goals  and  Benefits 

The  following  is  a  list  of  exit  criteria  for  the  LOCIS  program: 

a.  Enable  proactive  decision  making  by  commanders/supervisors. 

b.  Reduce  the  time  to  re-plan  during  a  crisis-action  contingency. 

c.  Reduce  staff  hours  to  produce  capability  and  historical  reports. 

d.  Reduce  time  required  to  assess  the  impact  of  operational  changes  on  wing/unit  level 
logistics. 

e.  Provide  improved  assessment  of  the  logistics  capability  of  units  to  support  tasked  combat 
missions. 

In  addition,  the  following  is  a  list  of  the  primary  goals  for  LOCIS  that  all  fall  within  the  support 
of  the  Air  Expeditionary  Forces,  Agile  Combat  Support  and  Joint  Vision  2020: 

a.  Push  timely  information  to  logistics  personnel. 

b.  Minimize  time  searching  for  the  "right"  information. 

c.  Reduce  the  need  to  interpret  data. 

d.  Create  customized  reports  and  presentations. 

e.  Predict  situations  and  give  users  enough  time  to  prevent  problems. 

f.  Analyze  alternative  courses  of  action  through  easy-to-use  “what  if’  simulations. 

g.  Support  dynamic  re-planning  using  real-time  thresholds  and  alerts. 

h.  Feed  better  information  to  theater-level  command  and  control  systems. 
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In  summary,  an  effective  LOCIS  system  will  provide  multiple  benefits  including  more  timely 
information,  minimized  search  time,  reduced  data  interpretation,  predictive  simulations,  and 
dynamic  re-planning.  Currently,  because  of  personnel-intensive  tasks,  many  decisions  made  by 
USAF  commanders  require  multiple  people  researching  and  pulling  data  from  a  variety  of 
sources  in  order  to  supply  the  required  information  for  decision-making.  For  daily  standup 
meetings,  for  example,  the  commander  may  have  a  staff  of  people  preparing  presentation  charts 
with  data  that  is  several  hours  old  by  the  time  the  meeting  is  conducted.  Through  LOCIS,  this 
picture  is  drastically  different.  Based  on  user  preferences,  in  day-to-day  situations  as  well  as 
during  time  critical  situations,  LOCIS  pushes  accurate  and  timely  information  to  the  user. 
Thresholds  can  be  set  on  a  variety  of  critical  data  elements  that,  when  surpassed,  cause  new 
views  of  critical  information  to  be  presented  to  the  user.  LOCIS  also  allows  the  user  to  change 
parameters  and  run  scenarios  to  predict  30  day,  60  day  and  other  future  look-ahead  simulations. 
In  addition,  at  the  touch  of  a  button,  information  is  disseminated  and/or  captured  for  reviewing  at 
the  next  standup  briefing.  All  combined,  LOCIS  will  feed  better  information  to  the  warfighter. 

4  Background 

With  the  changes  that  have  occurred  in  both  the  geopolitical  environment  and  the  United  States 
Air  Force,  operational  units  are  challenged  now  more  than  ever  to  maximize  use  of  all  available 
resources,  while  maintaining  a  very  demanding  operational  and  deployment  tempo.  Logistics 
personnel  have  discovered  that  the  concept  of  ‘Do  more  with  less ’  has  evolved  from  an  initial 
draw  down  philosophy  to  a  permanent  concept  of  operations.  As  a  result,  mission  success 
depends  upon  logistics  now  more  than  ever.  Despite  the  rapid  changes  in  logistics  requirements 
and  constraints,  the  supporting  legacy  information  systems  have  remained  somewhat  stagnant, 
and,  as  a  result,  are  less  effective  in  supporting  senior  logistics  decision  makers  and  field 
personnel. 

The  downsizing  of  the  force  and  the  reduced  funding  and  sparing  for  parts,  combined  with 
drastic  changes  in  world  power,  have  led  to  a  requirement  for  even  greater  control  over  available 
resources  within  the  wing.  Current  management  information  systems  for  wing-level  logistics  are 
slow  and  manpower-intensive,  often  requiring  specialists  for  interpretation  of  data  before 
decision  makers  can  make  quick  and  accurate  logistics  decisions.  Figure  2  identifies  the  LOCIS 
vision  to  automate  and  make  accessible  key  information  to  decision  makers. 

LOCIS  efforts  concentrate  on  the  direct  sortie  production  logistics  functions  of  aircraft 
maintenance,  munitions,  and  supply  (operations  support).  These  will  be  referred  to  hereinafter  as 
first  tier  or  direct  sortie  production  functions  as  shown  in  Figure  3.  Second  tier  functions  include 
such  traditional  logistics  support  functions  as:  fuel,  supply  (base  support),  transportation, 
logistics  plans,  and  contracting.  The  remaining  (third  tier)  support  functions  are  termed  ancillary 
and  include:  civil  engineering,  disaster  preparedness/air  base  operability,  services, 
communications,  security,  personnel/manpower,  medical,  operations,  weather,  and  intelligence. 
Through  expansion,  it  was  intended  that  LOCIS  would  eventually  cover  the  other  tiers  of 
information,  giving  the  logistics  decision  makers  the  complete  set  of  data  for  making  decisions. 
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Figure  3.  Logistics  Functional  Area  Tiers 


LOCIS  Research  and  Development  included  a  series  of  activities  including  a  Feasibility  Study 
and  Requirements  Definition  before  the  complete  R&D  program  began  with  three  spirals.  The 
following  section  summarizes  the  feasibility  study,  requirement  definition.  Spiral  1  and  Spiral  2. 

4.1  Phase  I  -  LOCIS  Feasibility  Study 

To  test  the  soundness  of  the  concept,  AFRL  initiated  a  Feasibility  Study  in  March  1995  to 
determine  whether  there  is  a  need  for  improved  accuracy  and  timeliness  in  the  flow  of  logistics 
information  to  the  wing-level  command  and  control  processes.  One  of  the  first  tasks  performed 
during  the  Feasibility  Study  was  to  document  the  high-level  goals  and  processes  envisioned  by 
Laboratory  personnel.  This  was  done  via  a  facilitated  session  to  document  the  LOCIS  vision. 


The  facilitated  session  results  identified  the  following  as  LOCIS  goals: 

a.  Increase  the  accuracy  and  understandability  of  information. 

b.  Identify  critical  logistics  information  and  control  mechanisms  necessary  to  accomplish 
tasks. 

c.  Provide  real-time  information  sequenced  to  support  specific  events. 

d.  Maximize  the  use  of  current  legacy  automated  systems  to  obtain  logistics  information. 

e.  Increase  availability  of  embedded  training  at  the  workstation  level. 

f.  Ensure  any  solutions  support  mobilizations. 

Based  on  initial  field  interviews  and  observations  collected,  AFRL  determined  that  there  is  a 
significant  need  for  more  accurate  and  timely  information  for  logistics  decision  makers. 

4.2  Phase  II  -  LOCIS  Requirements  Definition 

As  the  next  step,  AFRL  initiated  a  Requirements  Definition  effort  in  April  1996.  The  goals 
identified  during  the  Feasibility  Study  were  re-examined  and  expanded,  generating  the  following 
objectives  for  the  Requirements  Definition  effort. 

a.  Support  proactive  decision  making  at  squadron  and  wing  levels  through  the  use  of 
automated  thresholds,  filtering  and  dissemination  techniques  -  LOCIS  should  ensure 
that  logistics  decision  makers  are  provided  with  better  information  to  manage  their 
responsibilities  and  ensure  that  better,  more  accurate,  and  less  manpower  intensive 
information  is  provided  real-time  to  the  Wing  Operations  Center  (WOC). 

b.  Reduce  the  need  to  interpret  the  logistics  information  -  The  information  provided 
should  not  require  interpretation  by  decision  makers  in  the  WOC  in  order  for  it  to  be 
immediately  useable  by  the  wing  battle  staff. 

c.  Provide  seamless,  easy  access  to  logistics  information  from  legacy  systems  -  LOCIS 
data  should  also  provide  a  much  better,  more  reliable  set  of  real-time  data  to  feed  wing 
level  capability  simulations. 

d.  Eliminate  manual  data  collection  and  reporting  -  LOCIS  data  should  provide  a  more 
current  and  accurate  set  of  information  to  command  and  control  systems  at  and  above 
wing  level. 

During  this  phase,  observations  were  collected  from  various  user  organizations  and  translated 
into  requirements  by  subject  matter  experts  and  lab  personnel.  Over  1,800  observations  were 
collected  from  various  logistics  users  and  organizations  at  file  following  locations. 
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a.  Air  Force  Special  Operations  Command,  Hurlburt  Field  -  All  logistics  functions  and 
Command  Post. 

b.  3rd  Wing,  Elmendorf  AFB  -  All  logistics  functions  and  Command  Post. 

c.  1st  Fighter  Wing,  Langley  AFB  Command  Post. 

d.  162nd  Fighter  Squadron,  Ohio  Air  National  Guard  -  Maintenance  Officer,  Production 
Supervisor  and  Expediter. 

e.  US  Transportation  Command,  Scott  AFB  -  Plans  and  Analysis  (J5). 

f.  HQ  AMC,  Scott  AFB  -  XPY,  DOO,  LGXI,  LGA,  LGF,  LGRC  -  World  Wide  Tanker 
Airlift  Command  Post. 

g.  HQ  ACC,  Langley  AFB  -  All  logistics  staff  functions. 

h.  1st  Fighter  Wing,  Langley  AFB  -  All  logistics  functions. 

i.  347th  Wing,  Moody  AFB  -  69th  &  70th  Fighter  Squadrons  -  All  Logistics  functions. 

j.  52nd  Fighter  Wing,  Spangdahlem  AB  -  All  Logistics  Functions,  Wing  Vice  Commander, 
Operation  Group  Commander  and  Command  Post. 

k.  86th  Airlift  Wing,  Ramstein  AB  -  Logistics  plans  and  command  post. 

l.  HQ  USAFE,  Ramstein  AB  -  Logistics  staff  functions  (LGM,  LGX,  LGT). 

m.  HQ  17th  AF,  Sembach  AB  -  All  logistics  functions. 

4.3  LOCIS  User  Advisory  Groups 

In  addition  to  the  feasibility  study,  requirements  development,  and  user  comments,  a  Senior 
Logistics  Advisory  Panel  (SLAP)  and  LOCIS  User  Group  (LUG)  were  used  extensively  in  the 
development  spirals  of  LOCIS. 

4.3.1  Senior  Logistics  Advisory  Panel 

The  SLAP,  which  includes  senior-level  Air  Force  managers  and  officers/govemment  officials, 
was  formed  to  provide  guidance  and  direction  throughout  the  LOCIS  program.  The  following 
were  tasks  performed  by  the  SLAP: 

a.  Review  and  prioritize  requirements  for  each  of  the  three  phases  of  the  LOCIS  spiral 
development  demonstrations.  Provide  developers  with  a  list  of  prioritized  requirements 
from  “essential”  to  “possible  follow-on  work.” 
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b.  Ensure  requirements  traceability  is  maintained.  All  LOCIS  requirements  must  be 
traceable  to  actual  user  needs.  Ensure  LOCIS  interview  trips  are  covering  the  right 
warfighters,  based  on  the  command. 

c.  Ensure  LOCIS  capabilities  are  built  independently  of  organizational  structure.  Also, 
LOCIS  must  be  capable  of  supporting  peacetime,  deployed,  and  all  phases  of  the  Air 
Expeditionary  Force  (AEF)  support  efforts  from  generation  through  sustainment  and  re¬ 
constitution. 

d.  Build  a  transitioning  plan  for  the  successful  integration  of  the  LOCIS  effort  into  the 
operational  Air  Force. 

Membership  to  the  SLAP  kept  consistent  representation  from  the  following  organizations 
through  the  development  spirals:  ACC/LGX,  AFSOC/LG,  AMC/LGM,  AFLMA,  AETC/LG, 
USAF/ILM,  AFC2ISRC/A4  and  various  wing  Maintenance  Group  Commanders  (MXGs).  The 
first  meeting  of  the  SLAP  occurred  to  review  the  Spiral  1  demonstration  in  October  2000.  The 
final  SLAP  meeting  occurred  at  the  conclusion  of  Spiral  3  in  December  2002.  Complete  results 
from  this  meeting  can  be  found  in  the  Year  3  Operational  Architecture  Specification. 

4.3.2  LOCIS  User  Group 

The  LUG  consists  of  personnel  representing  the  roles  being  supported  for  each  spiral.  Spiral  1 
roles  consisted  of  the  Operational  Group  Commander  (OG),  Logistics  Group  Commander  (LG) 
and  their  staff;  this  was  expanded  in  Spiral  2  to  include  the  Senior  Maintenance  Officer;  and  in 
Spiral  3  to  include  the  Maintenance  Group  Commander  (MXG),  Aircraft  Maintenance  Squadron 
Commander,  Maintenance  Supervisor,  and  Aircraft  Maintenance  Unit  Supervisors/ 
Superintendent.  This  group  provided  critical  feedback  on  a  screen-by-screen,  capability-by- 
capability  basis.  The  LUG  first  met  in  April  2000  and  consistently  met  every  few  months 
during  critical  development  periods  of  each  spiral.  The  last  LUG  meeting  was  at  the  end  of 
Spiral  3  in  December  2002.  Complete  results  from  the  previous  LUG  meetings  can  be  found  in 
the  Operational  Architecture  document.  Summary  results  can  be  found  in  Appendix  B  of  this 
document. 

4.4  LOCIS  Spiral  1 

The  LOCIS  program  in  Spiral  1  leveraged  a  number  of  AFRL  technologies,  commercial 
companies  and  USAF  collaborators  to  provide  a  concise  and  effective  demonstration  of  LOCIS 
capabilities.  A  series  of  well-timed  and  well-attended  demonstrations  helped  the  user  base  to 
understand  LOCIS,  but  more  importantly,  to  envision  LOCIS  expansion  ideas  and  how  LOCIS 
could  help  them. 

The  initial  step  of  the  Spiral  1  was  to  validate  the  1800  comments  received  during  the 
requirements  study.  This  was  successfully  done  with  visits  to  Hurlburt  Field,  Luke  AFB  and 
Langley  AFB.  Not  only  were  the  1800  comments  confirmed,  but  an  additional  700  were  added. 
These  over  2500  comments  were  placed  into  a  database  and  used  to  create  the  193  LOCIS  system 
requirements  that  can  be  found  in  Appendix  C. 
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The  initial  design  and  development  of  the  Spiral  1  software  was  done  using  the  Unified 
Modeling  Language  (UML)  and  updated  in  each  spiral.  Four  sets  of  diagrams  were  developed  to 
define  and  document  the  mission  application  development  process.  First,  the  UML  Use  Case  and 
Class  diagrams  were  created  that  provided  all  the  information  needed  and  the  relationship  of  this 
information  based  upon  the  user  requirements  defined  by  the  operational  architecture.  Secondly, 
the  Activity  and  Sequence  diagrams  were  developed  which  documented  and  identified  the 
required  interaction  and  flow  of  each  software  module  in  support  of  the  Use  Case  and  Class 
diagrams.  All  four  sets  of  UML  diagrams  were  published  in  the  System  Requirements 
Specification  document.  From  these  diagrams,  each  of  the  software  modules  were  coded  and 
integrated. 

In  achieving  the  goals  of  the  mission  applications,  several  commercial  companies  stepped 
forward  and  offered  hardware  and  software  support.  Of  these,  Microsoft  was  the  most 
cooperative  by  giving  several  thousand  dollars  of  software  for  each  of  the  LOCIS  servers.  In 
addition,  Microsoft  attended  several  of  the  Spiral  1  LOCIS  meetings  and  held  several 
teleconferences  with  the  team  to  help  guide  understanding  of  the  implementation  of  their  tools. 
Neopoint  was  also  a  technology  partner.  They  brought  to  the  table  free  web  phones  for  analysis, 
discounts  on  all  purchased  web  phones  and  free  site  surveys  for  web  phone  service  coverage. 
Finally,  in  the  area  of  biometrics,  I/O  Systems  provided  free  software  for  their  proximity 
detectors  and  fingerprint  readers. 

A  series  of  Spiral  1  demonstrations  was  held  26-28  September  2000  in  the  Logistics  Technology 
Integration  Environment  (LOGTIE)  facility  at  WPAFB.  The  first  demonstration  was  for  the 
LUG,  the  second  was  for  the  SLAP,  and  the  third  was  for  collaborators  including  the  Defense 
Advanced  Research  Project  Agency  (DARPA),  the  AEF  Battlelab,  and  other  organizations 
within  AFRL.  In  addition  to  the  final  Spiral  1  demonstrations,  a  subset  of  the  LOCIS 
demonstration  was  presented  in  a  private  suite  at  the  Logistics  Officer's  Association  (LOA) 
Conference.  Several  key  USAF  logisticians  attended  the  private  demonstrations. 

Comments  received  during  the  demonstrations  were  positive  and  reinforced  the  need  to  work 
closely  with  other  programs.  Specific  programs  identified  were  Global  Combat  Support  System 
Integration  Framework  for  the  USAF  (GCSS-AF),  Enhanced  Maintenance  Operation  Capability 
(EMOC),  and  Air  Force  Portal,  which  laid  the  direction  for  the  future  LOCIS  spirals. 

The  success  of  LOCIS  in  Spiral  1  was  based  upon  a  managed  set  of  collaborations  with  other 
organizations.  The  following  is  an  acknowledgement  of  those  organizations  that  helped  make 
LOCIS  a  success  in  Spiral  1 : 

a.  Air  Force  Command  and  Control  Intelligent  Surveillance  Reconnaissance  Center 

b.  Air  Force  Special  Operations  Command 

c.  Electronic  Systems  Command  (ESC) 

d.  56th  Fighter  Wing  (FW)  -  Luke  AFB 

e.  PACAF  Interim  Command  and  Control  System 

f.  Enhanced  Maintenance  Operations  Capabilities 

g.  Air  Force  Portal 
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4.5  LOCIS  Spiral  2 


The  success  of  the  Spiral  1  demonstrations  and  results  from  the  LUG  were  fed  into  the  Spiral  2 
development.  In  addition,  these  results  and  the  recommendations  by  the  LOCIS  SLAP  led  to  the 
development  of  a  Block  Release  of  Spiral  1  capabilities  at  Hurlburt  Field.  Col.  Mueller, 
AFSOC/LG  and  a  SLAP  member,  offered  Hurlburt  Field  as  a  site  for  the  LOCIS  Block  Release, 
which  also  became  known  as  the  Hurlburt  Field  LOCIS  Living  Lab.  The  following  paragraphs 
summarize  the  activity  at  Hurlburt  Field. 

4.5.1  Hurlburt  Field  Living  Lab 

The  Hurlburt  Field  Block  Release  created  a  parallel  effort  during  Spiral  2  by  the  LOCIS  team  that 
included  both  the  creation  of  a  working  version  of  the  concepts  from  Spiral  1  and  the  creation  of 
new  conceptual  capabilities  for  Spiral  2.  The  Hurlburt  Field  Block  Release  was  vital  to  the 
development  of  LOCIS  since  it  gave  daily  warfighter  feedback  on  LOCIS  capabilities.  LOCIS 
continues  to  be  used  on  a  daily  basis  by  the  MXG  leadership  at  Hurlburt  Field. 

The  Hurlburt  Field  Block  Release  included  all  tasks  required  to  make  the  conceptual  ideas  of 
Spiral  1  into  real  working  capabilities  and  to  link  the  capabilities  to  live  data,  which  included 
interfaces  to  Hurlburt  Field's  Command  and  Control  Database  (C2DB)  and  the  Mission  Capable 
(MICAP)  Asset  Sourcing  System  for  Windows  Operating  System  (WinMASS). 

The  Block  Release  In-brief  was  held  5  November  2001  and  installation  began  the  next  day  after 
Hurlburt  Field  leadership  signed  the  final  Base  Communications  paperwork.  The  training  of 
Hurlburt  Field  users  was  initiated  26-30  November  2001.  The  Hurlburt  Field  User's  Manual 
(document  number  1004266)  was  developed  and  submitted  as  a  hardcopy  as  well  as  on-line 
through  the  LOCIS  application.  In  addition,  a  Hurlburt  Field  Block  Release  Administration  Plan 
was  delivered  to  assist  administration  of  LOCIS  by  users  at  Hurlburt  Field. 

The  LOCIS  Human  Factors  Integrated  Product  Team  (IPT)  led  three  evaluations  of  the  Hurlburt 
Field  Block  Release.  The  results  of  the  evaluations  can  be  found  in  Appendix  B  and  were  used 
for  planning  Spiral  3  activities. 

4.5.2  Spiral  2  Demonstration 

In  conjunction  with  the  development  and  installation  of  the  Hurlburt  Field  Living  Lab,  Spiral  2 
development  was  scheduled  to  complete  with  a  series  of  demonstrations,  including  the  LOA 
conference.  During  Spiral  2,  two  key  collaboration  efforts.  Air  Force  Portal  and  Point  of 
Maintenance,  occurred.  Work  with  Air  Force  Portal  resulted  in  the  users  at  Hurlburt  Field 
having  the  option  to  access  LOCIS  through  the  Air  Force  Portal.  Though  the  original  intent  was 
to  have  a  single  logon,  the  Air  Force  Portal  implementation  at  that  time  did  not  allow  passing  of 
authentication  to  another  application.  The  final  implementation  had  LOCIS  as  a  personal  link  for 
Air  Force  Portal  users  at  Hurlburt  Field. 

The  second  collaboration  was  with  the  Point  of  Maintenance  activity  at  Hurlburt  Field.  More 
specifically,  the  LOCIS  software  and  documentation  were  changed  to  portray  modifications  to 
work  orders  on  the  flightline  made  with  Point  of  Maintenance  hardware.  The  demonstration 
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simulated  closing  a  work  order  on  the  flightline,  which  causes  an  aircraft  icon  to  change  from  red 
to  green  on  the  LOCIS  display. 

The  main  focus  of  the  Spiral  2  development  was  to  tie  to  real  data.  This  was  accomplished  at 
Hurlburt  Field  by  linking  to  their  local  Maintenance  Operations  Center  (MOC)  database,  C2DB, 
as  well  as  the  WinMASS  for  Mission  Capable  (MICAP)  data.  In  addition  to  the  data  tie-in' 
Spiral  2  also  improved  several  of  the  user  interface  views  and  updated  the  alert/waming 
capability. 

The  biggest  change  in  Spiral  2  was  the  redesign  of  the  architecture  from  a  heavy  client 
distributed  system  to  a  thin  client  Java  2  Enterprise  Edition  (J2EE)  system  compliant  with  the 
evolving  GCSS-AF. 

5  LOCIS  Development  Process  and  Spiral  3 

Throughout  the  spiral  development,  the  following  key  areas  were  the  centers  of  activity: 
Operational  Architecture,  System  Architecture,  Mission  Applications,  Qualitative/Quantitative 
Reports,  and  Demonstrations.  The  following  paragraphs  describe  the  process  for  each  of  these 
areas  and  the  resulting  development. 

5.1  Operational  Architecture 

The  following  steps  were  used  to  produce  the  Operational  Architecture  for  each  spiral. 

a.  Develop  Scenario  Outline  of  LOCIS  spiral  demonstration. 

1.  Identify  requirements 

2.  Critical  data 

3.  Sources  of  data 

b.  Coordinate  with  Human  Factors  (HF)  for  functional  prioritization. 

c.  Determine  SLAP  and  LUG  Process. 

1 .  Identify  proper  composition  based  on  functionality 

2.  Identify  and  contact  members 

3.  Identify  process  for  small  groups  (work  with  HF  team) 

4.  Set  up  schedule 

5.  Identify  what  needs  to  be  posted  on  website 

6.  Determine  how  to  validate  information 

d.  Develop  requirements  process  (see  Figure  4). 

e.  Identify  Critical  Data,  Sources  of  Data  and  associated  Business  Rules. 

f.  Finalize  scenario  and  document  information. 

g.  Validate  scenario  and  information  with  SLAP  and  LUG. 

h.  Final  documentation  of  Operational  Architecture. 
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Figure  4.  Requirements  Process 


The  LOCIS  user  profiles  included  the  following  personnel:  Maintenance  Group  Commander, 
Operations  Group  Commander,  Fighter  Squadron  Maintenance  Supervisor  and  Superintendent. 

Through  each  spiral,  a  set  of  functional  capabilities  was  produced.  Based  upon  the  success  of 
these  capabilities  they  were  moved  forward  into  the  next  spiral  or  removed.  Figure  5  shows  the 
LOCK  Operational  Architecture.  Detailed  information  of  the  Operational  Architecture, 
including  minutes  from  the  SLAP  and  LUG  meetings,  can  be  found  in  the  Spiral  3  Operational 
Architecture  Document  (1007830);  see  Reference  Documents.  Figure  5  indicates  the  four  key 
LOCK  data  sources:  EMOC,  Operational  Data  Store  (ODS),  Core  Automated  Maintenance 
System  (CAMS),  and  Combat  Ammunition  System  -  Ammunition  Control  Point  (CAS-A). 
Agents  are  used  to  pull  information  into  the  LOCK  repository  where  it  is  fused  and  fimneled  to 
the  user  through  profile-based  web  pages.  The  Aircraft  Status  Reporting  Tool  (ASRT),  is  a 
passive  data  collection  tool  using  FM  radios  to  automatically  pull  critical  information  from  the 
base  RF  radio  system. 
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Figure  5.  Operational  Architecture 


In  Spiral  3,  the  following  were  the  key  functional  capabilities  that  leveraged  the  previous  two 
spirals: 

a.  Develop  print/print  preview/create  reports  capabilities. 

b.  Combined  flying  schedule  at  the  wing  level. 

c.  Redesign  the  wing  status  at-a-glance  to  reduce  or  eliminate  the  need  to  scroll. 

d.  Extend  Recap  to  include  data  prior  to  the  weekend  or  holiday. 

e.  Allow  Zulu  time  conversion/display. 

f.  Add  additional  timeline  customization. 

g.  Develop  781 A  view  which  displays  all  open  discrepancies  by  tail  number. 

h.  Add  additional  critical  data  to  the  roll-over  on  the  A/C  icon. 

i.  Address  several  Graphical  User  Interface  (GUI)  issues  including  resize  of  screen  and 
memory  of  column  size  and  location. 

j.  Add  additional  information  to  the  drill  down,  additional  data  to  the  MICAP  screens  and 
additional  data  to  the  supply  screens. 

k.  Add  a  weekly  calendar  inspection  view. 
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l.  Add  the  ability  to  see  the  status  of  line  numbers  and  be  notified  when  they  are  behind  in 
the  launch  sequence. 

m.  Develop  a  top  level  geographical  layout  of  a  base/aircraft  status. 

n.  Add  capability  to  drill  down  on  scheduled  and  actual  activity  on  the  flying  schedule. 

o.  Add  an  option  for  a  single  panel  workspace  with  the  current  dual  and  quad  options. 

p.  Develop  a  capability  to  create  thresholds  on  launch  sequence,  task  requirements,  and 
munitions  availability. 

q.  Add  capability  to  enhance  decision  support  functions  to  include  forecasting  capabilities. 

r.  Add  capability  to  forecast  munitions  availability  based  on  a  given  tasking  at  a  specific 
deployment  location. 

s.  Develop  78 1  -K  View  which  displays  all  delayed/deferred  discrepancies  by  tail  number. 
5.2  System  Architecture 

The  system  architecture  was  the  study,  analysis  and  design  of  Commercial-off-the-Shelf  (COTS) 
and  Govemment-off-the-Shelf  (GOTS)  products  and  the  architecture  created  by  these  products 
that  satisfied  the  operational  architecture’s  requirements. 

The  most  important  product  from  the  system  architecture  was  the  System  Requirements 
Specification  (SRS),  which  was  the  guiding  document  for  the  spiral  demonstration  development. 

During  Spiral  2,  the  architecture  from  Spiral  1  was  completely  redesigned.  The  Spiral  1 
architecture,  though  distributed  and  web-based,  involved  a  thick  client  requiring  multiple 
Microsoft  products  including  Microsoft  Excel,  Outlook  and  Digital  Dashboard.  This  architecture 
was  not  compliant  with  the  evolving  GCSS-AF,  which  began  to  move  to  the  emerging  J2EE. 
During  Spiral  3,  this  conversion  to  J2EE  was  completed.  Also  in  Spiral  3,  the  concept  of  web 
services  was  implemented  through  a  combination  of  J2EE  and  Microsoft.Net  Web  Services. 
This  created  an  open  environment  capable  of  supporting  J2EE  and  Microsoft  technologies.  The 
software  design  is  covered  in  more  detail  in  the  Mission  Application  section  of  this  document. 

The  system  architecture  was  also  responsible  for  identifying  any  additional  system  hardware  and 
software  required  for  the  demonstrations.  A  series  of  studies  during  each  spiral  was  performed. 
This  included  data  mining,  biometrics  and  wireless  studies  in  Spiral  1;  messaging,  portal,  pivot 
table/snapshot/links,  information  fusion,  data  mining,  trend  analysis,  simulation  and  dynamic 
rescheduling  studies  in  Spiral  2;  and  proactive  decision  support,  bioscience,  agents,  data 
manipulation,  architecture  and  framework  studies  in  Spiral  3.  More  specifically,  more  than  200 
slides  were  developed  for  the  Spiral  3  Preliminary  Design  Review  (PDR)  that  assessed  the 
usability  and  availability  of  technologies  for  LOCIS.  The  studies  and  associated  information  can 
be  found  in  the  specific  spiral  PDR/Critical  Design  Review  (CDR)  slides  as  well  as  the  Final 
Report  for  each  spiral.  Figure  6  depicts  the  core  technologies  that  have  been  the  theme  of  each 
LOCIS  spiral. 
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LOCIS  fuses  information  from  multiple  sources  in  order  to  provide  a  complete  decision  making 
capability.  Because  of  this,  LOCIS  itself  is  not  a  data  store  (neither  retaining  nor  storing 
information).  LOCIS  pulls  from  multiple  data  systems  to  fuse  elements  and  map  them  to  specific 
decision  support  views  within  LOCIS  based  upon  the  user  profile.  This  fusion  capability  will, 
among  other  things,  filter  data,  trigger  warnings  or  alerts  based  upon  new  values,  and  update  the 
LOCK  data  repository  appropriately.  Technologies  involved  in  this  area  include  data  fusion, 
access  to  data  warehouses,  filtering  and  agents. 


Therefore,  LOCK  is  dependent  on  the  current  USAF  information  systems  for  access  to  data. 
Specifically,  LOCK  is  required  to  reside  within  the  evolving  GCSS-AF  and  fuse  data  from  a 
variety  of  sources,  such  as  EMOC,  CAMS  and  CAS-A. 


5.2.1 .2  Proactive  Decision  Support 

LOCK  provides  proactive  decision  support  capabilities  in  a  users’  environment,  which  may 
include  a  wall  display,  desktop,  Personal  Digital  Assistant  (PDA),  or  cell  phone.  These 
capabilities  include  (1)  a  ”what-if'  predictive  capability,  (2)  a  best  tail  and  health  of  fleet 
advisory  tool  that  is  based  on  rules  and  simulation,  (3)  munitions  tracking  against  schedule 


14 


requirements  and  (4)  personal  alerts  and  warnings  of  key  indicators  that  are  set  to  individual 
thresholds.  These  tools  allow  the  user  to  make  better  decisions  regarding  current  situations,  but, 
more  importantly,  allow  the  user  to  better  prepare  and  better  assess  future  situations. 

5.2. 1.3  Dynamic  User  Interface 

The  key  to  the  presentation  and,  ultimately,  the  support  for  making  decisions  is  the  dynamic  user 
interface.  This  interface  is  customizable  by  each  user  through  the  use  of  web  page  options,  such 
as  data  refresh,  timeline  extensions  and  icon  symbols.  Once  the  user  tailors  these  views,  the  user 
can  then  combine  them  into  personalized  workspaces  that  can  be  quickly  reviewed  at  any  time. 
In  this  way,  the  user  can  very  quickly  assess  what  has  happened,  what  is  happening  and  the  key 
areas  where  decisions  are  required. 

In  addition,  these  workspaces  can  be  individually  tied  to  specific  alerts  and  warnings  such  that 
when  an  event  occurs,  key  workspaces  are  automatically  presented.  This  capability  saves  time  in 
searching  and  interpreting  information  during  critical  periods.  These  interface  features  are  based 
on  information  being  pushed  to  the  user,  which  in  turn  is  based  on  unique  user  preferences. 
Technologies  involved  with  developing  this  intelligent  user  interface  include  push  technology, 
agents,  and  advanced  visualization  techniques. 

5.2.2  System  Interfaces 

In  fulfilling  mission  requirements,  LOCIS  will  interface  with  the  following  systems,  receiving 
data  elements  as  indicated: 

CAMS  Core  Automated  Maintenance  System 

CAS-A  Combat  Ammunition  System-Ammunition  Control  Point 

EMOC  Enhanced  Maintenance  Operation  Capability 

ODS  Operational  Data  Store 

C2DB  Hurlburt  Field  Command  &  Control  Database 

Note:  All  databases  accessed  by  LOCIS  are  import  only— LOCIS  does  not  store  the  data  for 
historical  use.  LOCIS  is  not  a  data  store. 

5.3  Mission  Application  Development  and  Integration 

The  software  development  and  integration  of  the  three  key  technical  areas  of  LOCIS  (information 
fusion,  decision  support,  and  dynamic  user  interface)  occurred  within  the  mission  application 
development  and  integration. 

To  support  the  development  and  delivery  of  the  Mission  Application,  a  series  of  deliverables 
were  created  and  submitted  to  the  LOCIS  customer.  The  following  documents  should  be 
referenced  for  more  detailed  information  regarding  the  Mission  Application:  Demonstration 
Plan,  Software  Product  Specification,  Training/Administrator’s  Plan,  Software  End  Item,  and 
Software  Version  Description — see  Reference  Documents. 
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Figure  7  identifies  the  software  capabilities  as  they  evolved  throughout  the  three  spirals.  At  the 
top  (in  parentheses)  is  the  main  emphasis  of  each  spiral.  As  shown,  the  proactive  decision 
support  and  the  dynamic  user  interface  were  the  key  development  areas  in  Spiral  1.  Spiral  2 
upgraded  the  user  interface  and  fused  real  data  from  Hurlburt  Field  data  sources,  but  did  not  add 
much  technology  in  the  decision  support  area.  Spiral  3  focused  on  standardizing  data  access 
from  USAF  data  sources  and  adding  munitions  capability  to  the  system. 


Spiral  1  Spiral  2  Spiral  3 

(Maintenance)  (Supply)  (Munitions) 


Dynamic 
User  l/F 

•  Web-enabled 

•  Client  Server 

•  Thick  Client 

•  Auto  status  reports 

•  Web-centric 

•  Thin  Client 

•  Customized  Interface 

•  Workspace  Wizard 

•  Adaptive  Interface 

•  Automatic/Ad  Hoc 
Report  Generator 

Proactive 

Decision 

Support 

•  Simulation 

•  Concept  Wamings/Alerts 

•  Integrated  A/C  Status  & 
Schedules 

•  Pivot  Table  Concept 

•  Triggers 

•  Robust  and  Concept 
Wamings/Alerts 

•  Automated 

Scheduling 

Recommendations 

•  Capability 
Assessments 

•  Extended  Agent 
Framework 

Information 

Fusion 

•  Simulated  data 

•Agent  Monitoring  and 

Feed  Data 

•  16  SOW  MOC  C2DB: 
Maintenance  and 

Flying  Schedule  Data 

•  WinMASS:  MICAP 
status 

•  Agent  Monitoring 
and  Data  Feed 

•  ODS  &  CAMS 
data  access 

•  EMOC  data 
access 

Figure  7.  Development  of  Spiral  Software  Capabilities 

The  key  focus  of  the  Spiral  3  Mission  Application  development  was  the  modularization  and 
standardization  of  the  software.  In  particular,  the  goal  was  to  create  a  more  robust  LOCIS  that 
could  also  be  installed  from  base  to  base  economically.  The  key  to  this  was  the  development  of  a 
more  modular  architecture  as  shown  in  Figure  8.  This  figure  highlights  the  Java  servelets  and 
applets  that  were  used  by  the  web  server  to  produce  the  thin  client  interface.  This  architecture 
conforms  to  the  evolving  GCSS-AF  Integration  Framework.  Figure  8  also  depicts  the  COTS 
tools,  Formula  One  and  JRules,  for  the  report  generation  and  advisory  tool  capability. 
Additionally,  Figure  8  defines  the  internal  warning  and  alerts  thresholds  (WAT),  housekeeping, 
preference,  message  and  trigger  managers  that  manipulate  the  data  received  from  the  variety  of 
data  sources. 

A  number  of  studies  were  performed  with  results  being  presented  at  the  Spiral  3  PDR  and  CDR. 
These  studies  included  the  following:  report  generation  tools,  rule-based  tools,  web  servers  and 
databases.  Final  study  results  chose  Formula  One  for  report  generation,  JRules  for  rule-based 
tools,  BEA  Weblogic  as  the  web  server  and  Oracle  as  the  database.  The  main  driver  for  the 
Oracle  choice  was  the  USAF  license — Spiral  1  and  2  used  Microsoft  SQL  Server,  but  for  future 
deployment  economics  it  was  decided  that  Oracle  was  a  better  choice. 
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Figure  8.  LOCIS  Architecture 

To  enable  the  goals  of  Spiral  3,  particularly  the  modularity  and  standardization  goals,  several 
changes  were  made  in  the  three  technical  areas.  The  following  is  a  summary  of  the  key  mission 
application  activities  in  these  three  areas. 

5.3.1  Information  Fusion 


The  fusion  of  information  was  a  key  focus  for  Spiral  2  and  again  in  Spiral  3.  In  particular,  for  the 
Hurlburt  Field  Block  Release,  LOCIS  required  access  to  real-time  information  via  the  AFSOC 
C2DB.  However,  with  the  requirement  to  be  more  standard  in  Spiral  3,  interfaces  to  ODS, 
EMOC,  CAMS  and  CAS-A  were  built.  In  addition  to  these  data  sources,  the  Electronic  Data 
Warehouse  (EDW)  was  also  pursued.  However,  after  multiple  discussions  with  EDW  personnel 
it  was  decided  that  currency  of  data  in  EDW  was  such  that  it  was  not  valuable  to  LOCIS  users. 

To  perform  the  CAS-A  interface,  DSI  joined  the  BAE  Systems  team  bringing  their  key  munitions 
knowledge.  Working  closely  with  the  LOCIS  team,  DSI  developed  a  Web  Service  interface  to 
munitions  data  from  CAS-A  which  could  be  called  from  LOCIS  for  near-real  time  updates  of 
munitions  data  and  analysis  based  on  that  data. 

To  perform  the  CAMS  interface,  a  GFE  tool,  developed  by  KLSS,  called  CAMS  Connectivity 
Development  Tool  (CCDT)  was  integrated  and  used. 

EMOC  and  ODS  interfaces  were  developed  through  standard  Oracle  XML  mapping  and 
extraction  techniques. 
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5.3.2  Decision  Support 


Decision  support  in  Spiral  2  leveraged  the  success  of  Spiral  1  in  the  area  of  (1)  notifications  of 
alerts  and  warnings,  (2)  user-definable  thresholds,  and  (3)  “what-if  ’  simulation.  The  “what-if  ’ 
capability  being  provided  by  the  KLSS  Scaleable  Integration  Model  for  Objective  Resource 
Capability  Evaluations  (SIMFORCE)  tool.  In  Spiral  3,  the  alerts  and  warnings  were  further 
enhanced  along  with  a  plan  for  two  new  capabilities:  a  surge  or  deployment  analysis  tool  and 
munitions.  The  advisory  tool  was  to  leverage  the  SIMFORCE  “what-if’  tool  and  the  JRules  rule- 
based  advisory  tool.  However,  scheduled  priorities  were  such  that  the  advisory  tool  was 
developed  in  concept  only.  The  following  paragraphs  summarize  the  Spiral  3  decision  support 
capabilities: 

a.  Alerts  and  Warnings  -  Spiral  3  changed  the  paradigm  of  the  alerts  and  warnings.  No 
longer  did  the  user  need  to  go  to  a  particular  screen  to  set  and  monitor  alerts.  Each  screen 
was  enabled  with  an  alert  button  much  like  each  screen  was  given  a  print  capability.  The 
alert  button  opened  a  window  of  options  to  add/remove  and  update  the  data  element  or 
situation  to  be  monitored  by  the  alert.  This  included  A/C  availability  alerts  brought 
forward  from  Spirals  1  and  2,  deployment,  munitions,  and  launch  sequence  alerts,  and 
new  munitions  availability  alerts. 

b.  Surge  and  Deployment  Planning  tool  -  The  biggest  challenge  to  Spiral  3  was  the  tool  to 
assess  the  surge  and  deployment  planning.  Within  this  tool  were  embedded  capabilities 
that  included  Health  of  Fleet,  Munitions  analysis,  system  integrity  and  algorithms  that 
leveraged  user-defined  evaluation  factors.  In  the  end,  the  final  product  was  an  extremely 
useful  planning  tool  that  allowed  the  user  to  be  alerted  when  an  element  of  the  plan  was 
exhibiting  behavior  that  could  jeopardize  it.  The  result  is  both  a  planning  tool  and  a 
predictive  monitoring  tool  that  performs  background  checks  in  providing  a  proactive 
capability. 

c.  Asset  Forecast  -  The  asset  chosen  in  Spiral  3  for  this  capability  was  munitions. 
Algorithms  behind  the  view  calculate  asset  availability  from  the  current  date  forward  up 
to  thirty  days.  The  results  are  shown  visually  in  daily  green/yellow/red  squares  indicating 
predictions  of  whether  the  available  munitions  could  meet  the  planned  requirements  for  a 
particular  day.  Green  indicates  fully  met,  yellow  partially  met,  and  red  not  met  at  all. 
This  view  could  be  used  to  track  any  asset  forecast.  Also  note  that  this  screen  has  an 
"alert  me"  button  that  allows  a  user  to  be  notified  if  availability  is  going  to  be  difficult  in 
X  number  of  days,  as  defined  by  the  user.  See  Figure  9  for  the  Munitions  Forecast  View. 
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Figure  9.  Munitions  Forecast  View 

One  view  within  the  deployment  planning  capability  is  shown  in  Figure  10.  This  is  the  Health  of 
Fleet  view  that  defines  the  critical  evaluation  parameters,  their  user-defined  level  of  importance 
(weight)  and  the  tail  number  ranking.  Very  quickly  it  is  possible  for  the  user  to  note  the  critical 
activities  that  will  come  due  before  or  during  the  proposed  deployment,  surge  or  other  scheduling 
event.  It  is  possible  on  this  screen  to  change  the  weight  of  each  activity  and  then  re-calculate  the 
tail  ranking.  Other  views  within  the  deployment  planning  capability  are  the  system  integrity 
table  and  the  munitions  evaluation  table,  which  allows  the  user  to  map  munitions  for  the 
upcoming  activity.  Like  all  LOCIS  activity,  this  capability  was  designed  by  working  closely  with 
the  user  and  providing  solutions  to  their  needs. 
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The  dynamic  user  interface  development  in  Spiral  3  leveraged  the  work  from  Spiral  1  and  2.  A 
number  of  operational  capabilities  were  updated  based  upon  user  comments  and  several  new 
capabilities  were  added.  Throughout  the  LOCIS  spirals,  human  factors  played  a  critical  role  in 
the  development  of  views  and  collecting  user  feedback.  This  was  particularly  true  for  the 
development  of  the  status  at-a-glance  view,  which  contained  the  combined  view  of  status  and 
schedule  as  well  as  supply  information.  The  original  design  in  Spiral  1  was  well  received  and 
updated  in  Spiral  2.  However,  one  key  problem  was  the  ability  to  view  the  full  wing  on  one 
screen.  This  was  a  key  comment  received  from  multiple  users.  To  address  this  issue,  the  LOCIS 
team  added  this  requirement  to  the  other  user  interface  requirements  and  made  minor 
modifications  to  the  screens.  One  modification  was  a  compression  button  that  allowed  the  user 
to  compress  all  icons  to  one  screen.  This  changed  the  format  of  the  screen  but  allowed  the  user 
to  choose  his  or  her  personal  viewing  preference. 


A  new  capability  developed  in  Spiral  3  is  shown  in  Figure  11.  This  is  the  geographic  view  that 
maps  parking  locations  via  GPS  coordinates  to  an  aerial  map  of  the  base  (the  base  pictured  is 
Langley  AFB).  To  allow  the  user  to  zoom  in  and  out  required  icons  to  be  dynamically  resizable. 
This  required  a  complete  redesign  of  the  icon.  However,  human  factors  ensured  that  the  icon 
behavior  is  the  same  on  all  pages,  meaning  that  the  mouse-over  and  drill  downs  are  identical. 
This  is  just  one  case  where  human  factors  played  a  key  role  in  the  development  and  evolution  of 
LOCIS  views. 
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Figure  11.  LOCIS  Geographic  View 

Another  area  that  has  evolved  over  the  three  spirals  was  the  concept  of  a  workspace.  By  Spiral  3 
this  concept  had  become  similar  to  a  "Favorites"  folder  when  surfing  the  Internet,  but  also  having 
the  ability  to  combine  views  into  a  favorite  selection.  For  example,  single,  dual  and  quad  panels 
are  available  to  be  created  by  user  choice  and  saved  under  an  individual  user's  workspace.  These 
views  are  filled  with  any  of  the  over  thirty  views  available  within  LOCIS.  The  views  in  a 
workspace  are  active,  live  panels  for  which  the  look,  feel,  and  behavior  are  identical  to  the 
individual  web  pages  from  which  they  are  chosen.  Figure  12  shows  such  a  web  page.  It  is  a  dual 
panel  showing  supply  aircraft  and  related  supply  drivers  on  the  left  panel  and  the  status  of  all  the 
aircraft  on  the  right  panel.  This  is  similar  to  the  view  that  is  used  most  often  by  users  at  Hurlburt 
Field. 
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Figure  12.  Dual  Panel  Workspace 


5.4  Collaboration 

As  in  previous  spirals,  LOCIS  collaborated  with  a  number  of  organizations  in  Spiral  3. 

a.  Air  Force  Command  and  Control  Intelligence  Surveillance  Reconnaissance  Center  - 
AFC2ISRC  has  been  the  key  sponsor  of  LOCIS  throughout  the  contract.  Their  support 
and  attendance  at  PDRs,  CDRs,  and  demonstrations  has  been  important  to  the  success 
LOCIS  achieved. 

b.  Air  Force  Special  Operations  Command  at  Hurlburt  Field  -  The  support  and  development 
of  the  LOCIS  Living  Lab  at  Hurlburt  Field  has  been  a  key  relationship  that  has  allowed 
AFRL  to  test  LOCIS  in  a  living  lab  environment  and  to  capture  vital  user  feedback. 

c.  Air  Combat  Command  (ACC)  -  The  ACC  at  Langley  had  offered  themselves  as  the  Field 
Test  for  LOCIS  and  has  been  instrumental  in  providing  feedback  during  the  LOCIS 
development. 

d.  Air  Force  Portal  -  Air  Force  Portal  has  supported  LOCIS  and  even  offered  to  place  the 
LOCIS  demonstration  on  their  portal.  The  Spiral  2  software  was  delivered  to  AF  Portal 
in  the  spring  of  2002.  In  addition,  a  data  mapping  between  ODS  and  LOCIS  was 
delivered  at  the  end  of  Spiral  3. 
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5.5  Demonstrations 


To  validate  the  LOCIS  efforts,  it  is  important  to  demonstrate  the  concepts  each  year  to  the  users. 
The  demonstrations  for  Spirals  1  and  2  were  identified  in  the  background  section  of  this 
document.  The  final  demonstration  for  Spiral  3  was  held  at  the  LOGTIE  facility  at  Wright 
Patterson  AFB.  The  demonstration  was  held  in  conjunction  with  the  LUG  meeting  and  was 
attended  by  the  following  personnel : 


Name 

Maj  Mark  Gray 

1st  Lt  Michael  Carawan 


Organization 


AFC2ISRC/LG 


3 8 8MOS/MXOO 


CMSgt  Robert  Boyd  HQACC/LGXI 

CMSgt  Darrell  Bridges  HQPACAF/LGMM 

CMSgt  Joe  Moran  33AMXS/MAZ 

SMSgt  Stephen  McLaughlin  33AMXS/MXAAO 

SMSgt  Andrew  Smith  1 AMXS/MAXP 

MSgt  Leonard  Stolirchick  1EMS 

TSgt  Bruce  Moore  16MXG/CDP 

TSgt  Brian  Sternberg  AFSOC/LGMX 


In  addition  to  the  LUG  demonstration,  a  demonstration  was  given  to  a  number  of  key  LOCIS 
collaborators.  The  attendance  list  for  this  meeting  is  as  follows: 


Ronald  Wright 

MSG/ESS 

Edwin  Allen 

AFRL/VACD 

Henry  Gifford 

MSG/ESS 

Todd  Trickier 

MSG/ESS  (BTAS) 

Rick  Cowan 

MSG/ESS 

Randy  Koram 

AFMC/XP-AO 

Louis  Scacca 

AFRL/HES 

Gary  Hardenburg 

BAE  SYSTEMS 

Clark  Moskop 

BAE  SYSTEMS 

Rob  Roy 

DSI 

Deb  Mitta 

GTRI 

Shana  Shelton 

MSG/ESD  (EDW) 

Dave  Erickson 

GTRI 

Gary  Smith 

AFRL/PRTA 

Cheryl  Holtz 

AFRL/PRTA  (UTC) 

JB  Schroeder 

AFRL/VACC 

Barbara  Masquelier 

AFRL/HESR  (Chief) 

Christopher  Curtis 

AFRL/HESR  (LOCIS  PM) 
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A  third  and  final  demonstration  was  given  to  the  SLAP  members.  Those  attending  this  meeting 
were  as  follows: 


Name 

Organization 

Col  Peaches  Kavanaugh 

AFC2ISRC/LG 

Col  Steve  Cooper 

AETC,  12th  LG/CC 

Col  Charles  Williams 

AFSOC/LG 

Col  Ronnie  Mercer 

AFLMA/CC 

Col  Karl  Lewandowski 

AMC/LGM 

Col  Richard  Berry 

16thMXG/CC 

Lt  Col  Kenneth  Grimes 

ACC/LGXI 

Maj  Mack  Breeland 

USAF/ELMM 

Ms.  Tricia  Gober 

SSG/ILM 

Mr.  Dan  Kugel 

AFRL/XPH 

Col  (Ret)  Steve  Powers 

KLSS 

Col  (Ret)  Bob  Johnson 

CACI 

Col  (Ret)  Robert  Me  Gill 

CACI 

CMS  (Ret)  Clark  Moskop 

BAE  SYSTEMS 

CMS  (Ret)  Dick  Weimer 

KLSS 

Mr.  Gary  Hardenburg 

BAE  SYSTEMS 

Mr.  Chris  Curtis 

AFRL 

Ms.  Barbara  Masquelier 

AFRL 

To  support  these  demonstrations,  a  Demonstration  Plan  (CDRL  A010)  was  developed  and 
delivered.  In  addition,  to  assist  in  preparing  and  performing  the  demonstration,  a 
Training/Administrator’s  Plan  (CDRL  A012)  was  developed  and  delivered.  Reference  either  of 
these  documents  for  more  specific  information  on  the  demonstration  hardware,  software  and 
scenario — see  Appendix  D. 

5.6  Qualitative  and  Quantitative  Reports 

It  is  very  important  to  the  LOCIS  sponsor  that  the  research  performed  and  the 
products/capabilities  created  are  what  the  end  user  wants.  To  meet  this  goal,  several  reports  were 
generated  each  year  and  used  to  measure  the  success  of  LOCIS.  More  specifically,  a  transition 
plan,  Cost  Benefit  Analysis  (CBA)  and  final  report  of  user  feedback  and  comments  were  created. 
The  transition  plan  for  Spiral  2  was  delivered  in  the  Technology  Transition  Report  (CDRL 
A016).  The  CBA  and  feedback/comments  can  be  found  in  Appendix  A  and  in  Appendix  B. 
respectively.  In  addition,  this  final  report  captures  a  summary  of  the  Spiral  3  effort. 
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5.7  Program  Management 


BAE  SYSTEMS  managed  the  LOCIS  program  from  a  series  of  documents  including  the 
Program  Plan  of  Execution  (PPofE),  which  was  tailored  specifically  for  the  LOCIS  development 
effort.  The  PPofE  is  available  from  the  Data  Accession  List  (CDRL  A008).  The  Data  Accession 
List  also  includes  the  following  documents:  Operational  Architecture  Specification,  Software 
Development  Plan,  Style  Guide  for  the  C  Language,  and  Style  Guide  for  the  Java  Language. 

The  LOCIS  success  was  assisted  by  the  use  of  a  website  with  an  area  accessible  to  everyone  and 
a  password  protected  area  for  developers,  customers  and  users.  The  website  address  is 
www.program-support.com/LOCIS/index. 

6  Summary 

The  LOCIS  program  in  Spiral  3  has  continued  to  expand  capabilities — but  more  importantly,  the 
program  has  remolded  the  architecture  to  promote  an  economical  base-to-base  deployment.  This 
has  been  accomplished  by  creating  a  complete  thin  client  application  and  a  modular  architecture. 
In  addition,  the  data  source  interfaces  have  been  rewritten  to  standard  USAF  sources  including 
CAS-A,  CAMS,  EMOC  and  ODS.  The  results  have  been  extremely  well  received  from  the  users 
at  Hurlburt  Field  and  from  numerous  decision  makers  across  the  USAF.  Spiral  3  has  increased 
the  cost  benefits  for  LOCIS  as  can  be  seen  in  the  Cost  Benefit  Analysis  in  Appendix  A. 

The  CBA  and  also  user  comments  have  met  many  of  the  exit  criteria.  A  field  test  that  is  being 
planned  will  be  the  final  test  of  the  exit  criteria.  However  the  following  items  can  be  noted: 

a.  The  alert/waming  working  autonomously  provides  a  proactive  decision  support 
capability. 

b.  Though  only  used  in  day-to-day  scenarios,  the  fused  information  on  the  multiple  screens 
should  reduce  time  to  replan  during  a  crisis-action  contingency. 

c.  The  CBA  proves  reduced  staff  hours. 

d.  The  CBA  and  also  user  comments  affirm  the  reduction  of  time  to  assess  the  input  of 
operational  changes. 

e.  Though  only  used  in  day-to-day  scenarios,,  initial  assessments  predict  improved 
assessment  to  support  tasked  combat  missions. 

The  Hurlburt  Field  Living  Lab  and  the  multiple  spiral  demonstrations  have  proven  that  LOCIS 
stayed  focused  on  the  needs  of  the  warfighter  throughout  the  program's  development. 
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Appendix  A  —  Spiral  3  Cost  Benefit  Analysis 

Introduction 

This  appendix  describes  the  methodology  and  presents  the  framework  used  to  develop  this  CBA. 
The  purpose  of  the  CBA  is  to  document  tangible  and  intangible  cost  benefits  of  LOCIS  for  Air 
Force  operational  and  logistics  command  and  control  decision-makers.  Examples  of  intangible 
costs  are:  standard  Web  GUI  across  the  Air  Force,  accurate  data  from  legacy  systems, 
intuitiveness  of  standard  interface  and  the  cost  of  blue  suit  manpower  (technicians)  returning  to 
their  primary  jobs  back  on  the  flight  line. 

The  methodology  is  based  on  DoD  and  commercial  practices,  and  has  been  tailored  to  fit  the  Air 
Force.  One  example  of  this  is  the  usage  of  the  term  “Cost  Avoidance.”  Cost  Avoidance  refers  to 
the  release  of  technicians  back  to  their  primary  duties.  There  are  no  realized  dollar  savings, 
rather  better  use  of  manpower  resources.  Due  to  the  increased  reduction  in  manpower  in  the  Air 
Force,  references  to  cost  savings  are  carefully  evaluated  because  these  costs  will  be  removed 
from  an  organization’s  operating  budget.  This  action  reduces  manpower  that  was  never  funded 
to  perform  the  additional  duties  outside  their  Air  Force  Specialty  Code  (AFSC)  duties.  It  has 
been  made  veiy  clear  to  the  team  during  all  data  collections  that  there  are  not  enough  personnel 
to  maintain  current  aircraft.  For  these  reasons,  “Cost  Avoidance”  terminology  is  used.  LOCK 
has,  and  will  continue  to  demonstrate  that  core  capabilities  allow  wing  and  squadron  decision¬ 
makers  to  do  their  job  faster  and  more  efficiently.  This  will  allow  them  to  perform  other  critical 
tasks  not  being  performed  today  due  to  a  lack  of  time.  In  addition,  when  slots  can  be  reduced,  in 
particular  for  data  gathering  and  reporting,  it  is  envisioned  that  these  personnel  would  return  to 
performing  their  primary  AFSC  of  maintaining  aircraft  and  related  support  equipment. 

The  basic  premise  of  the  methodology  is  to  evaluate  cost  benefit  from  multiple  viewpoints.  The 
most  common  viewpoints  are  the  cost  benefit  related  to  an  organization’s  labor  and  product.  The 
product  viewpoint  has  not  been  evaluated.  The  primaiy  product  produced  and  enabled  by  LOCK 
are  decisions,  which  are  very  subjective,  especially  when  they  are  converted  to  cost  benefit. 
However,  to  remain  compliant  with  the  original  premise  of  multiple  viewpoints,  cost  benefits 
were  created  by  role,  OG,  LG,  Deputy  Operations  Group  Command  Maintenance  (DOG/MA), 
Deputy  Logistics  Group  Commander,  Chief  Operations  Group,  Chief  Logistics  Group,  Sortie 
Generation  OIC,  Sortie  Generation  Chief,  MOC  Controller,  MOC  Senior  Controller  and 
Production  Super  and  tied  to  a  capability,  reporting  take  offs,  identifying  trends,  reporting 
completed  tasks  or  updating  the  status  board  and  by  LOCK  requirements  accomplished  in  Spiral 
3.  Each  of  these  is  discussed  in  more  detail  in  the  following  paragraphs. 

The  framework  used  to  calculate  the  cost  benefits  is  through  spreadsheets.  This  provides  key 
management  personnel  on  the  program  the  ability  to  incorporate  updates  and  changes  as  required. 
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The  following  definitions  are  provided  to  clarify  the  techniques  utilized  to  develop  the  CBA. 

a.  Functional  Process  Improvement  -  The  application  of  a  structured  methodology  to  define 
a  function’s  “as  is”  environment,  its  objectives  and  strategy  for  achieving  those 
objectives,  and  a  program  of  incremental  improvements  made  through  functional, 
technical,  and  economic  analysis  and  decision  making.  DoD  8020. 1-M,  Interim 
Management  Guidance  on  functional  Process  Improvement  documents  this  program. 

b.  Activity  Based  Costing  -  A  technique  that  identifies  the  costs  of  producing  the  primary 
products  and  services  for  a  business  area. 

c.  Economic  Analysis  -  A  technique  that  provides  for  the  systematic  comparison  of 
alternate  approaches  to  problem-solving,  using  financial  analysis  and  risk  analysis 
algorithms.  Individual  costs  and  benefits  are  associated  with  alternative  investment 
opportunities,  taking  into  account  the  lifecycle  characteristics  of  each  investment. 

d.  Cost  Avoidance  -  Cost  Avoidance  refers  to  the  release  of  technicians  back  to  their 
primary  duties.  There  are  no  realized  dollar  savings,  rather  better  use  of  manpower 
resources.  This  is  a  component  of  the  Economic  Analysis. 

e.  Return  On  Investment  -  The  amount  of  time  required  to  pay  for  the  savings  documented 
in  a  business  case  or  Functional  Economic  Analysis  (FEA).  This  is  a  component  of  the 
Economic  Analysis. 

f.  Functional  Economic  Analysis  or  Cost  Benefit  Analysis  -  A  well-documented  decision 
package  that  supports  a  proposed  expenditure  of  investment  funds  along  with  a  plan  of 
actions.  It  is  the  final  result  of  using  all  of  the  tools,  techniques,  methodologies,  tactics 
and  strategies  of  the  FPI  Program. 

The  numbers  and  results  presented  herein  are  estimates  developed  by  functional  personnel  based 
on  information  collected  from  user  interviews.  Again  these  numbers  only  apply  to  the  roles, 
capabilities  and  requirements  demonstrated  in  Spiral  3  and  the  block  release.  The  roles  used  in 
the  CBA  include  the  OG,  LG,  DOG/MA,  Deputy  Logistics  Group  Commander,  Chief  Operations 
Group,  Chief  Logistics  Group,  MOC  Controller,  MOC  Senior  Controller,  and  Production  Super. 
In  addition,  the  numbers  represent  the  cost  benefit  for  a  wing  consisting  of  three  squadrons. 

Summary  and  Conclusions 

Table  A.1  represents  the  reduction  in  labor  hours,  cost  avoidance  and  return  on  investing  from 
the  viewpoints  of  role  and  capabilities,  and  requirement  as  they  related  to  the  LOCIS  Spiral  3 
demonstration. 
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Table  A.1  —  Spiral  3  CBA  Summary 


LOCIS  CBA:  Spiral  3  Summary 

i 

Percentage  Reduction  in  Labor  Hours  Per  Wing  (3  Squadrons) 

By  Role 

51% 

By  Requirement 

51% 

Cost  Avoidance  Per  Year  Per  Wing  (3  Squadrons) 

By  Role 

$ 

1,701,928 

By  Requirement 

$ 

771,942 

Cost  Avoidance  Per  Month  Per  Wing  (3  Squadron) 

By  Role 

$ 

141,827 

By  Requirement 

$ 

64,329 

LOCIS  Cost  (Year  1  Only) 

$ 

454,017 

Return  on  Investment  (in  Months)  (Recuring  costs  not  included.) 

By  Role 

3 

By  Requirement 

7 

The  results  are  presented  as  ranges  and  should  be  viewed  as  estimates.  Keep  in  mind  that  a 
benefit  is  something  of  value  produced  by  an  alternative  solution.  It  is  the  Return  On  Investment 
(ROI).  All  potential  benefits  must  be  quantified  or  they  cannot  be  used  in  the  analysis.  For 
example,  the  reduction  in  labor  hours  are  depicted  as  51%,  however  a  more  accurate  view  would 
be  to  expect  roughly  a  50%  reduction  in  the  labor  hours  required  for  activities  enabled  by  LOCIS. 
This  is  also  true  for  the  ROI.  The  ROI  is  depicted  to  occur  in  3-7  months;  however,  a  more 
accurate  portrayal  of  this  is  that  it  occurs  in  less  than  12  months.  DOD  8020. 1-M  states  that  an 
investment  is  warranted  if  the  ROI  occurs  within  three  years.  It  is  important  to  note  that  the  ROI 
is  expected  to  shorten  as  additional  capabilities  are  demonstrated  in  the  upcoming  spirals. 

Standard  formulas  were  used  in  the  calculations  to  determine  each  benefit  listed  below: 

Reduction  in  Labor  Hours  =  1  -  (Future  Labor  Hours  with  LOCIS  /  Current  Labor  Hours 
without  LOCIS) 


Cost  Avoidance  =  Current  Labor  Cost  without  LOCIS  -  Future  Labor  Cost  with  LOCIS 


Return  On  Investment  =  Installation/Implementation  Costs  /  Cost  Avoidance  Per  Month 
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Cost  Assumptions 


The  primary  components  that  drive  costs  are  labor  and  implementation,  support  and  sustainment. 
Table  A.2  presents  manpower  costs  for  the  roles  supported  in  Spiral  1.  Costs  are  presented  in 
yearly,  monthly,  weekly,  daily  and  hourly  amounts.  The  numbers  above  the  Manpower  Cost 
headings  were  used  to  calculate  the  corresponding  amount  for  each  role.  In  addition,  the  number 
of  workdays  per  year  and  per  month  are  presented  and  used  to  calculate  activity  per  month  and 
activity  per  year  for  each  viewpoint. 

Table  A.2  -  Manpower  Assumptions 


LOCIS  CBA:  Assumptions 

12 

52 

219 

1752 

Manpower  Cost 

Salary 

Monthly 

Weekly 

Daily 

Hourly 

06  -  Colonel:  LG 

$  112,458 

$ 

9,372 

$ 

2,163 

$ 

514 

$ 

64 

05  -  Lt  Col.:  DLG 

$  86,989 

$ 

7,249 

$ 

1,673 

$ 

397 

$ 

50 

03  -  Capt:  SGF  OIC 

$  64,416 

$ 

5,368 

$ 

1,239 

$ 

294 

$ 

37 

E9  -  Chief:  Chief  LG,  MOC  Sen  Ctrlr,  Chief  MSgt, 

Pro  Supr,  SGF  Chief 

$  63,116 

$ 

5,260 

$ 

1,214 

$ 

288 

$ 

36 

E6  -  Tech  Sgt:  MOC  Controller 

$  40,630 

$ 

3,386 

$ 

781 

$ 

186 

$ 

23 

i  _ 

Number  of  Work  Days  Per  Month 

18.25 

Number  of  Work  Hours  Per  Year 

1752 

29 


Table  A.3  presents  the  costs  associated  with  the  installation,  implementation,  support  and 
sustainment  of  LOCIS.  Included  in  these  costs  are  the  costs  of  hardware,  software  and  labor. 
Labor  costs  have  been  calculated  based  on  the  number  of  man-months  times  the  cost  per  man- 
month.  Travel  has  been  included  for  installation  and  training. 


Table  A.3  —  LOCIS  Spiral  3  Installation,  Implementation, 
Sustainment,  Support  Costs 


Installation/Implementation  Costs  Per  Wing  (3  Squadron  Wing  with  similar  MDSs) 

Item 

Sustainment 

Cost 

Man-Months 

Cost  Per  Man-Month 

(Recurring) 

Material 

LOCIS 

Hardware  Server  (Dell  PowerEdge  4400) 
Software 

$ 

10,018 

NA 

NA 

Oracle  DB  Server 

$ 

4,599 

NA 

NA 

Weblogic 

$ 

17,000 

Formula  One 

$ 

19,950 

FM  -  NET  (ASRT) 

Hardware 

$ 

6,900 

NA 

NA 

Software 

$ 

3,300 

NA 

NA 

Labor 

LOCIS 

Installation  &  Testing 

$ 

180,000 

3 

$20,000 

Training 

$ 

30,000 

0.5 

$20,000 

Support 

$ 

60,000 

3 

$20,000 

20k  Per  Yr 

FM  Net 

Vocabulary 

$ 

30,000 

2 

$5,000 

Installation 

$ 

10,000 

1 

$10,000 

Training 

Support 

$ 

$ 

5,000 

24,750 

0.5 

1.5 

$10,000 

$5,500 

8K  Per  Yr 

SIMFORCE 

Data  Setup  (Failure  &  Resource  Analysis) 

$ 

30,000 

5 

$6,000 

Training 

Support 

$ 

$ 

15,000 

7,500 

0.5 

1 

$10,000 

$2,500 

2.5K  Per  Yr 

$ 

454,017 

Viewpoint  1  -  By  Role  and  Capability 


Table  A.4  presents  major  capabilities  grouped  by  role.  The  foundation  of  this  CBA  lies  in  the 
difference  between  similar  work  accomplished  without  LOCIS  (or  currently)  and  with  LOCIS. 
LOCIS  functional  expert’s  (LUG  and  SLAP  members)  comments,  plus  the  observations  from  the 
many  field  visits  conducted,  identified  the  amount  of  time  spent  performing  each  capability  each 
day  and  the  number  of  people  within  each  role  performing  the  task.  These  components  drive  the 
costs  that  calculate  effort  expended  for  a  major  capability.  They  are  multiplied  together,  along 
with  the  labor  costs  for  the  applicable  role  to  achieve  a  cost  by  activity.  Labor  hours  and  costs, 

per  month  and  per  year,  are  calculated  using  the  days  per  month  and  per  year  identified  in 
Table  A.2  -  Manpower  Assumptions. 
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Table  A.4  -  Spiral  3  Labor  and  Costs  By  Role  and  Capability 


|  LOCI S  CBA:  By  Role  ”  | 

Wing  Activity  By  Roles 

Current  (w/o  LOCIS) 

Time 

(Hours  per  Number  of 

Day  per  People  Per 

Person)  Day  Cost 

Future  (with  LOCIS) 

Time 

(Hours  per  Number  of 

Day  per  People  Per 

Person)  Day  Cost 

LG,  06,  Colonel 

Monitor  Aircraft  Status 

2 

1 

$128 

1 

1 

$64 

Monitor  Flying  Schedule 

1 

1 

$64 

0.5 

1 

$32 

Problem  Notifications 

1 

1 

$64 

0.5 

1 

$32 

Solution  Alternatives 

2 

1 

$128 

1 

1 

$64 

Deputy  LG,  05,  Lt.  Col. 

Monitor  Aircraft  Status 

2 

1 

$99 

1 

1 

$50 

Monitor  Flying  Schedule 

1 

1 

$50 

0.5 

1 

$25 

Problem  Notifications 

1 

1 

$50 

0.5 

1 

$25 

Trend  Analysis 

1.5 

1 

$74 

0.5 

1 

$25 

Solution  Alternatives 

2 

1 

$99 

1 

1 

$50 

Chief  LG,  E9,  CMSgt 

Monitor  Aircraft  Status 

2 

1 

$72 

1 

1 

$36 

Monitor  Flying  Schedule 

2 

1 

$72 

1 

1 

$36 

Monitor  Munitions  Status 

0.5 

1 

$18 

0.2 

1 

$7 

Problem  Notifications 

2 

1 

$72 

1 

1 

$36 

Solution  Alternative 

2 

1 

$72 

1 

1 

$36 

Sortie  Gen  OIC,  03,  Capt,  Sortie  Gen  Fit 

Chief,  E8,  SMSgt 

Monitor  Aircraft  Status 

2 

6 

$441 

1 

6 

$221 

Select  A/C  for  Deployment 

2 

6 

$441 

0.5 

6 

$110 

Monitor  Flying  Schedule 

2 

6 

$441 

1 

6 

$221 

Problem  Notifications 

1 

6 

$221 

0.5 

6 

$110 

Trend  Analysis 

1 

6 

$221 

0.5 

6 

$110 

Solution  Alternative 

1 

6 

$221 

0.5 

6 

$110 

MOC  Controller,  E6,  Tech  Sgt;  Sen 

Controller,  Prod  Super,  E7, 

MSgt;Munition$  NCO,  E7,  MSgt 

Slide  Generation  (Standup) 

1 

21 

$487 

0.5 

21 

$243 

Aircraft  Status  Recording  (FM  Net) 

4 

21 

$1,948 

1 

21 

$487 

Monitor  Aircraft  Status 

6 

21 

$2,922 

4 

21 

$1,948 

Select  A/C  for  Deployment 

1 

21 

$487 

0.5 

21 

$243 

Monitor  Flying  Schedule 

6 

21 

$2,922 

4 

21 

$1,948 

Monitor  Munitions  Status 

3 

6 

$417 

1 

6 

$139 

Problem  Notifications 

4 

21 

$1,948 

2 

21 

$974 

Solution  Alternative 

4 

21 

$1,948 

2 

21 

$974 

Total 

60 

$16,129 

29.7 

$8,357 

Activity  Per  Month 

Activity  Per  Year 

1,095 

13,140 

* 

$294,346 

$3,532,152 

542 

G,504 

$152,519 

$1 ,030,224 
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Viewpoint  2  -  By  Requirement 

Table  A.5  presents  labor  hours  and  costs  by  requirements.  LOCIS  functional  experts  identified 
the  amount  of  time  spent  performing  each  requirement  each  day  and  the  number  of  people 
performing  the  task. 

Intangible  Benefits 

In  addition  to  benefits  derived  from  specific  measurements,  LOCIS  also  provides  a  number  of 
intangible  benefits. 

a.  Accurate  data  from  legacy  systems. 

b.  Standard  web  GUI  across  Air  Force. 

c.  Intuitive  interface. 

d.  Manpower  not  accounted  for  within  the  government  is  considered  intangible  (e.g., 
support  of  systems,  data  collection,  managing  MOC  boards). 

e.  Improved  data  accuracy  of  C2  systems. 

f.  Increases/improves  decision  time  of  wing  leadership. 

g.  Real  time  data  display. 

h.  Saves  on  lost  sorties. 

i.  Saves  management  from  needing  to  reconfigure  an  aircraft  multiple  times. 

j .  Reduction  in  the  number  of  late  takeoffs. 
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Table  A.5  -  Spiral  3  Labor  and  Costs  By  Requirement 


LOCIS  CBA:  By  Requirement  | 


Rqt 

Nbr 

Title 

Current  (w/o  LOCIS) 

Nbr  of 

Time  Per  Day  People  Per 

(Hrs)  Day  Cost 

Time  Per 
Day  (Hrs) 

Future  (With  LOCIS) 

Nbr  of 

People 

Per  Day  Cost  Notes 

o 

DLG 

Chief  LG 

MOC 

Moc  Sen 

SGF  QIC 

2  c 
£  i 

o  £ 
a 

1 

Identify  User 

0.1 

8 

$32 

0.1 

8 

$32 

Sys  Admin 

IN 

11111 

1  1 

2 

Identify  Job  Responsibility 

0.1 

B 

$32 

Sys  Admin 

D 

11111 

1  1 

3 

Identify  Position  and  User  Access  Restrictions 

0.1 

8 

$32 

Sys  Admin 

ID 

11111 

1  1 

4 

Identify  Subject  /  Topic 

0.1 

8 

$32 

WrkSpc  Mod 

i 

11111 

1  1 

5 

Identify  Default  Data  Elements 

0.1 

8 

$32 

Wrk  Spc  Mod 

i 

11111 

1  1 

6 

Identify  Default  Data  Changed  by  User 

0.1 

8 

$32 

Wrk  Spc  Mod 

i 

11111 

1  1 

7 

Identify  Placement  of  Data 

0.1 

8 

$32 

Wrk  Spc  Mod 

i 

11111 

1  1 

8 

Define  Critical  Resources 

0.5 

8 

$159 

0.1 

8 

$32 

i 

11111 

1  1 

9 

Define  User  Thresholds 

0.1 

8 

$32 

i 

11111 

1  1 

13 

Establish  Available  A/C  Threshold 

0.1 

8 

$32 

i 

11111 

1  1 

21 

Establish  Abort  Rate  Threshold 

0.1 

2 

$9 

1  1 

22 

Establish  Repeat/  Recur  Rate  Threshold 

0.1 

2 

$9 

1  1 

23 

Define  Munitions  Thresholds 

0.5 

2 

$43 

0.1 

2 

$9 

1  1 

24 

Define  Supply  Thresholds 

0.1 

4 

$16 

1  3 

40 

Receive  Weekly  Flying  Schedule 

0.5 

8 

$159 

0.1 

6 

$32 

i 

11111 

1  1 

43 

Define  Vehicle  Consumption  Standards  Requirements 

0.5 

2 

$30 

0.3 

2 

$18 

1  1 

44 

Define  POL  Consumption  Standards 

0.5 

2 

$30 

0.3 

2 

$18 

1  1 

45 

Define  Wing  Aircraft  System  Configuration 

0.7 

2 

$50 

0.3 

2 

$22 

1  1 

46 

Define  AGE  Standards 

0.5 

2 

$30 

0.3 

2 

$18 

1  1 

48 

Project  Future  Events  /  Requirements 

1 

1 

$50 

0.5 

1 

$25 

1 

49 

Determine  How  Shortfall  Can  Be  Rectified 

2 

4 

$319 

0.7 

4 

$112 

i 

1 

2 

50 

Determine  if  the  Unit  Can  Support  A  Requirement 

1 

2 

$86 

0.5 

2 

$43 

1  1 

51 

Identify  Timing  Associated  with  supporting  a  Threshold 

1 

2 

$87 

0.2 

2 

$17 

i 

1 

53 

Adjust  Future  Mission  Requirements 

2 

3 

$247 

1 

3 

$123 

i 

1 

1 

55 

Collect  Specified  Threshold  Actual  Data 

Transparent 

i 

1111 

56 

Compare  Actuals  to  Thresholds 

Transparent 

i 

1111 

58 

Identify  Negative  Trends  (Timing) 

1 

8 

$318 

0.5 

8 

$159 

i 

11111 

1  1 

59 

Identify  NMCS  Status 

0.5 

5 

$84 

0.25 

5 

$42 

1  1  1 

1  1 

60 

Identify  PMCS,  PMCM,  PMCB  A/C 

0.5 

5 

$84 

0.25 

5 

$42 

1  1  1 

1  1 

61 

Identify  Available  A/C  Location 

0.5 

9 

$150 

0.25 

9 

$75 

2  1  1 

1  4 

62 

Identify  A/C  Configuration 

2 

4 

$237 

1 

4 

$118 

2 

2 

63 

Identify  Available  Types 

1 

2 

$59 

0.5 

2 

$30 

1 

1 

Identify  Available  Munitions  Locations  (On-Site  and  Off- 

64 

Site  Storage,  etc.) 

2 

3 

$218 

0.5 

3 

$54 

1  1  1 

65 

Identify  Type  Available  and  Quantify 

1 

2 

$59 

0.5 

2 

$30 

1 

1 

66 

Identify  Available  Components  (On-hand  or  In  Supply) 

0.5 

2 

$30 

0.3 

2 

$18 

1 

1 

67 

Assign/Identify  Available  Tail  Numbers  To  Mission 

2 

3 

$247 

1 

3 

$123 

i 

1 

1 

68 

Assign/Identify  Available  Munitions  to  Mission 

2 

3 

$247 

1 

3 

$123 

i 

1 

1 

69 

Determine  Shortfalls  between  tasks  and  resourceas 

1 

8 

$317 

0.5 

8 

$159 

i 

1111 

3 

70 

Identify  NMCS/MICAP,  NMCM,  NMCB  A/C 

0.5 

5 

$84 

0.25 

5 

$42 

1  1  1 

1  1 

71 

Identify  A/C  Location 

0.5 

5 

$84 

0.25 

5 

$42 

1  1  1 

1  1 

72 

Identify  Scheduled  Maintenance 

0.5 

2 

$30 

0.25 

2 

$15 

1 

1 

73 

Identify  Real-Time  Sortie  Generations  A/C  Discrepancies 

1 

10 

•  $390 

0.75 

10 

$292 

i 

11111 

1  3 

74 

Identify  A/C  Scheduled  for  Maintenance  Problems 

1 

8 

$317 

0.5 

8 

$159 

i 

1111 

3 

86 

Identify  TPFDD  Tasking  Problems 

2 

2 

$175 

1 

2 

$87 

i 

1 

93 

Identify  Weather  Problems 

1 

5 

$182 

0.75 

5 

$136 

1  1  1 

1  1 

96 

Identify  Problems  with  Shift  Assignments 

2 

3 

$218 

1 

3 

$109 

1 

1  1 

106 

Prioritize  Problem 

2 

5 

$383 

0.5 

5 

$91 

1  1  1 

1  1 

108 

Identify  Sources  of  Supply 

0.5 

4 

$66 

0.25 

4 

$33 

1  1 

1  1 

109 

Identify  Quantity  On-Hand 

0.5 

6 

$109 

0.25 

6 

$54 

11111 

1 

110 

Identify  Parts  Arrival  Time 

0.5 

6 

$109 

0.25 

6 

$54 

11111 

1 

114 

Identify  Date  Of  Arrival 

0.5 

7 

$127 

0.1 

7 

$25 

11111 

1  1 

115 

Right  Line  Requisitions  Part 

0.5 

5 

$84 

0.3 

5 

$50 

1  1  1 

1  1 

118 

MOC  Pushes  Highlighted  A/C  Button  Problem  Spelled  Out 

0.2 

4 

$26 

0.1 

4 

$13 

1  1 

1  1 

119 

LOCIS  Users  See  Options  for  A/C 

0.6 

5 

$118 

0.2 

5 

$39 

i 

1  1 

1  1 

121 

LOCIS  Moves  Original  A/C  Into  Maintenance  Area 

0.5 

5 

$84 

0.25 

5 

$42 

1  1  1 

1  1 

122 

LOCIS  Moves  New  A/C  Into  Original  Flying  Line 

0.5 

5 

$84 

0.25 

5 

$42 

1  1  1 

1  1 

175 

Report  Loads  Begin 

0.6 

7 

$151 

0.3 

7 

$76 

3 

1  3 

176 

Report  Completed  Munitions  Load 

0.6 

7 

$151 

0.3 

7 

$76 

3 

1  3 

177 

Report  POL  Status 

0.5 

4 

$66 

0.2 

4 

$26 

1  1 

1  1 

178 

Report  Pilot  Show  At  A/C 

0.5 

7 

$127 

0.25 

7 

$63 

11111 

1  1 

179 

Report  Maintenance  Actions  Status 

0.5 

8 

$145 

0.25 

8 

$72 

11111 

1  2 

180 

Report  A/C  Taxi 

0.5 

7 

$127 

0.25 

7 

$63 

11111 

1  1 

181 

Report  A/C  Take-off 

0.5 

7 

$127 

0.25 

7 

$63 

1111*1 

1  1 

182 

Report  A/C  In-Right  States 

0.2 

3 

$25 

0.1 

3 

$12 

i 

1 

1 

184 

Report  A/C  Location 

0.5 

5 

$105 

0.25 

5 

$52 

i 

1111 

189 

Report  Land  Times 

0.2 

4 

$24 

0.1 

4 

$12 

2 

2 

190 

Capture  Actual  Data 

0.5 

7 

$127 

0.25 

7 

$63 

11111 

1  1 

Total 

44.70 

$7,192 

21.85 

$3,667 

Activity  Per  Month  816  $131,257  399  $66,929 


Activity  Per  Year  _ 9,769 _ $  1,575,085  4,765 _ $  603,143 


Note:  Number  of  People  Per  Day  column  includes  LG,  OG,  Deputy  LG,  Deputy  OG,  Chief  LG,  Chief  OG,  MOC 
Controller,  and  MOC  Chief. 
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SGF  Chief 

Pro  Sup 


Appendix  B  -  LOCIS  Usability  Evaluation  Results 

This  appendix  documents  results  of  the  LOCIS  usability  evaluations  conducted  during  Spirals  1, 
2  and  3.  During  Spiral  2,  a  series  of  evaluation  sessions  were  conducted  to  assess  the  specific 
user  interface  features  incorporated  into  the  LOCIS  Block  Release  capability.  Also  during  Spiral 
2,  a  single  evaluation  session  was  conducted  in  conjunction  with  the  Spiral  2  Demonstration. 
During  Spiral  3,  a  single  usability  evaluation  was  conducted  in  conjunction  with  the  Spiral  3 
Demonstration. 

B.l  Spiral  1  SLAP/LUG  Results 

LOCIS  Usability  Demonstrations  were  held  on  26-27  September  2000  in  the  AFRL/HESR  LOG- 
TIE  Facility  at  Wright-Patterson  Air  Force  Base.  The  purpose  of  these  demonstrations  was  to 
solicit  user  inputs  on  the  set  of  LOCIS  capabilities  demonstrated  for  Year  1.  Two  usability 
demonstrations  were  conducted — one  for  members  of  the  Senior  Logistics  Advisory  Panel  (held 
on  27  September)  and  one  for  members  of  the  LUG  (held  on  26  September).  Members  of  the 
Senior  Logistics  Advisory  Panel  provided  LG/OG-level  perspectives.  Their  primary  objectives 
were  to  assess  demonstrated  LOCIS  capabilities  from  a  senior  management-level  perspective,  as 
well  as  recommend  strategies  for  future  transition  efforts.  Members  of  the  Logistics  User  Group 
offered  an  LG/OG  staff  perspective  on  the  usability  aspects  of  demonstrated  capabilities,  but  they 
also  offered  their  insights  on  potential  LG/OG  use  of  LOCIS. 

In  this  appendix,  feedback  obtained  from  each  user  group  is  documented.  A  comparison  of  the 
comments  attributed  to  each  group  is  offered  first.  In  comparing  the  feedback  across  groups,  we 
identified  a  number  of  common  themes,  and  these  themes  are  highlighted  in  paragraph  B.1.1.  In 
paragraphs  B.1.2  and  B.1.3,  the  two  sets  of  user  input  data  are  considered  individually.  These 
paragraphs  provide  more  detailed  documentation  on  specific  comments  offered  by  the  SLAP  and 
LUG  members  in  response  to  a  set  of  human  factors  questions  developed  for  each  group’s 
usability  demonstration.  In  paragraph  B.l. 4,  a  brief  discussion  outlining  lessons  learned  and 
recommendations  for  future  demonstrations  is  provided. 

Paragraphs  B.1.2  and  B.1.3  focus  on  SLAP  and  LUG  feedback,  respectively.  Note  that  in  these 
paragraphs,  participant  comments  are  characterized  according  to  one  of  four  LOCIS  capabilities 
demonstrated  during  the  Year  1  Demonstration. 

a.  Data  Management 

b.  Dynamic  User  Interface 

c.  Proactive  Decision  Support 

d.  Real-Time  Updating  of  Information 

During  the  Year  1  usability  demonstrations,  each  of  these  aspects  was  assessed  through  a  series 
of  questions.  Questions  in  a  given  series  were  designed  to  focus  participants’  comments  on  a  set 
of  supporting  features.  In  paragraphs  B.1.2  and  B.1.3,  comments  associated  with  a  given  LOCIS 
capability  are  further  characterized  according  to  the  set  of  features  supporting  that  capability. 
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B.1.1  Comparison  of  SLAP  and  LUG  Data 

As  described  in  the  Year  1  Demonstration  Plan,  objectives  of  the  two  data  collection  sessions 
were  different.  The  motivation  for  collecting  data  from  SLAP  members  was  to  capture  feedback 
directly  from  those  users  who  were  specifically  targeted  for  the  Year  1  effort.  One  of  our 
fundamental  design  goals  is  to  push  the  right  information,  to  the  right  people,  at  the  right  time. 
From  a  human  factors  perspective,  achieving  this  goal  is  possible  only  when  we  “know  the  user.” 
By  interviewing  SLAP  members  within  the  context  of  the  LOCIS  Year  1  Demonstration,  we 
were  able  to  collect  feedback  representative  of  our  intended  user  group.  Inputs  (from  a  senior 
management-level  perspective)  on  the  potential  usefulness  and  appropriateness  of  demonstrated 
capabilities  and  technologies,  as  well  as  recommendations  for  future  transition  efforts,  were 
solicited.  While  the  intent  of  the  SLAP  data  collection  session  was  to  capture  top-level 
assessments  on  the  applicability  of  capabilities  demonstrated  for  Year  1,  SLAP  members  were 
equally  willing  to  provide  more  detailed  inputs  addressing  interface  design  and  usability. 

The  data  collection  session  conducted  for  LUG  members  was  designed  to  solicit  detailed 
assessments  of  LOCIS  capabilities,  where  the  focus  was  on  interface  design  and  usability.  In 
their  assessments,  LUG  members  considered  content  and  layout  of  information  views; 
technologies;  design  of  the  LOCK  web  page,  including  the  Alerts  and  Warnings  window  and  the 
design  of  multiple-view  workspaces;  and  interaction  strategies  required  for  control  of  LOCK 
information,  including  the  drill-down  capability,  PowerPoint  snapshot  feature,  and  the  book 
marking  feature.  In  some  cases,  they  also  used  their  insights  to  anticipate  LG/OG  reaction  to 
LOCK  features. 

Table  B.l  summarizes  the  types  of  data  collected  across  the  two  user  groups.  User  feedback  was 
categorized  according  to  one  of  four  demonstrated  capabilities:  data  management,  dynamic  user 
interface,  proactive  decision  support,  and  real-time  information  updating.  Once  categorized 
according  to  capability,  comments  were  further  classified  according  to  the  features  supporting 
that  capability. 

An  x  in  the  column  SLAP  DATA  ( LUG  DATA)  indicates  that  data  specific  to  a  given  feature  was 
collected  from  the  SLAP  group  (LUG  group).  As  indicated  in  Table  B.l,  the  data  collection 
team  was  able  to  capture  data  from  both  user  groups  for  a  majority  of  the  capability/feature  pairs. 
In  spite  of  the  differences  in  focus  and  the  level  of  detail  associated  with  comments  from  each 
group,  we  identified  several  common  themes.  These  themes  will  be  addressed  in  the  following 
paragraphs. 

Both  groups  concurred  on  the  issue  of  deployability.  Specifically,  they  emphasized  that  LOCK 
must  be  a  deployable  capability.  Security  was  another  issue  of  concern  to  the  two  user  groups — 
an  issue  they  felt  the  LOCK  team  should  continue  to  address  during  subsequent  years.  SLAP 
users  were  concerned  at  the  security  risks  associated  with  misplaced  proximity  cards,  and  they 
identified  cell  phone  vulnerability  to  jamming  as  an  area  of  concern.  LUG  members  also 

identified  security  as  an  issue  for  proximity  card  use.  As  did  SLAP  participants,  LUG  members 

identified  jamming  as  a  vulnerability  of  cell  phone  and  RF  communications  capabilities. 


35 


Data-related  issues  were  of  concern  to  both  groups — namely,  sources  of  data,  how  LOCIS  will 
accommodate  for  instances  in  which  data  is  inaccessible,  and  data  timeliness.  SLAP  members 
agreed  that  LOCIS  must  have  a  standalone  capability  in  those  instances  when  the  LAN  is  not 
operational  and  data  is  inaccessible.  Both  groups  recommended  incorporating  a  time  stamp  on 
LOCIS  data  screens.  SLAP  members  recommended  the  inclusion  of  a  time  stamp  on  all 
PowerPoint  snapshots.  Ultimately,  data  issues  will  be  of  particular  concern  during  the  Year  2 
effort,  and  those  raised  by  the  SLAP  and  LUG  will  be  addressed. 


Table  B.l.  Types  of  Feedback  Collected  across  User  Groups 


DEMONSTRATED 

CAPABILITY 

SUPPORTING  FEATURE 

SLAP 

DATA 

LUG 

DATA 

DATA  MANAGEMENT 

Snapshot 

X 

X 

Book  Marking 

X 

X 

Pivot  Table 

X 

X 

Drill  Down  Capability 

X 

DYNAMIC  USER  INTERFACE 

User  Preferences 

X 

Integrated  Flying  Schedule  and 

Aircraft  Status  Information 

X 

X 

Drill-Down  Data 

X 

Multiple-View  Workspaces 

•  X 

DEMONSTRATED 

CAPABILITY 

SUPPORTING  FEATURE 

SLAP 

DATA 

LUG 

DATA 

Wing  Capability  Indicator 

X 

PROACTIVE  DECISION 

SUPPORT 

Alerts  and  Warnings 

X 

X 

User-Definable  Thresholds 

X 

X 

What-If  Simulation 

X 

X 

REAL-TIME  INFORMATION 
UPDATING 

Voice-Based  Status  Changes 

X 

X 

Notification  of  Alert  and  Warning 
Messages 

X 

X 

Both  groups  expressed  an  interest  in,  as  well  as  a  need  for,  customizing  information  displays  to 
accommodate  for  individual  preferences. 

SLAP  and  LUG  members  identified  “holes”  in  the  list  of  thresholds  displayed  in  the  Threshold 
Settings  view,  and  both  groups  suggested  a  number  of  thresholds  for  potential  inclusion.  While 
both  groups  were  receptive  to  the  concept  of  user-definable  thresholds,  LUG  members  expressed 
more  enthusiasm  for  the  capability. 

In  considering  the  voice-based  status  update  capability,  SLAP  and  LUG  members  concurred  on 
the  need  for  some  type  of  feedback  mechanism  in  which  the  MOC  acknowledges  and  verifies 
status  changes. 
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Both  groups  identified  a  need  for  improving  the  manner  in  which  users  are  notified  of  incoming 
LOCIS-specific  e-mail  messages.  Their  comments  suggest  a  need  for  users  to  be  cued  upon 
arrival  of  such  messages.  They  also  suggest  a  need  for  these  role-based  e-mail  messages  to  be 
readily  distinguishable  from  alert  and  warning  messages.  One  recommendation  from  the  SLAP 
was  to  highlight  LOCIS-specific  e-mail  messages.  LUG  participants  suggested  two  separate 
mailboxes — one  for  role-specific  e-mail  messages  and  the  second  for  alert/waming  messages. 

LUG  member  insights  regarding  LG/OG  perceptions  and  reactions  proved  to  be  extremely  valid. 
Interestingly,  for  each  instance  in  which  they  were  asked  to  “predict”  how  SLAP  members  might 
react  to  various  LOCIS  features,  they  correctly  anticipated  SLAP  member  reaction.  The 
following  accurate  “predictions”  were  made. 

a.  LG/OG  use  of  the  “what-if  ’  simulation  capability  will  be  driven  primarily  by  individual 
preferences  and  management  style. 

b.  LG/OG  users  will  not  be  enthusiastic  about  adding  a  cell  phone  to  their  set  of  portable 
communications  devices. 

c.  Radios  are  viewed  as  the  most  important  mobile  communications  device.  Notifications 
of  real  emergencies  will  likely  come  via  radio  and/or  cell  phone.  LG/OG  users  will  not 
be  waiting  for  the  LOCIS  Alerts  and  Warnings  window  to  notify  them  of  real 
emergencies. 

d.  LG/OG  users  will  be  receptive  to  a  capability  allowing  them  to  capture  and  save 
snapshots  of  data — as  supported  by  LOCIS’  PowerPoint  capture  feature. 

e.  Not  all  LG/OG  users  will  be  receptive  to  using  LOCIS’  book  marking  feature  (providing 
links  to  real-time  data)  to  support  stand-up  meetings. 

f.  Current  LOCIS  displays  will  provide  too  much  detail  for  most  LG  and  OG  users. 

g.  Configuration  codes  should  be  more  generic.  (Some  SLAP  members  felt  configuration 
data  would  be  unnecessary  for  the  LG.) 

B.1.2  Senior  Logistics  Advisory  Panel  Data 

Inputs  from  the  SLAP  focused  primarily  on  senior  management-level  assessments  of 
demonstrated  LOCIS  capabilities  (i.e.,  appropriateness  and  applicability),  as  well  as 
recommendations  for  future  transition  efforts.  However,  SLAP  participants  were  also  willing  to 
address  interface  usability  and  design  issues,  particularly  as  they  related  to  features  associated 
with  OG  and  LG  default  workspaces.  Participant  comments  were  characterized  according  to  one 
of  the  four  LOCIS  capabilities  demonstrated  during  the  Year  1  demonstration:  data  management, 
dynamic  user  interface,  proactive  decision  support,  and  real-time  updating  of  information.  Each 

of  lliesc  capabilities  was  assessed  lliiuugli  an  iufuiiiial  questiuii-and-answci  session  held  aflci  (lie 

demonstration.  In  some  instances,  participants  also  provided  written  comments  relevant  to  issues 
raised  during  the  post-demonstration  discussions.  Verbal  feedback  obtained  during  the  question- 
and-answer  sessions,  as  well  as  written  comments,  is  documented  here. 
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Summary 


Issues  of  primary  importance  to  the  group  as  a  whole  included  deployability,  back-up  capability, 
sources  of  data,  and  security.  The  consensus  was  that  LOCIS  must  be  deployable  (i.e., 
operational  on  laptops),  and  participants  observed  it  would  be  a  “nonstarter”  if  it  does  not  work 
with  AEF.  Participants  agreed  that  LOCIS  must  have  a  standalone  capability  in  those  instances 
when  the  LAN  is  not  operational.  A  back-up  capability — specifically,  auto-archiving  onto  a 
designated  hard  drive — was  also  recommended.  With  respect  to  server  and  LAN  requirements, 
some  members  expressed  concern  over  the  insufficiency  of  existing  bandwidth  at  individual 
bases  and  emphasized  LOCIS  must  become  part  of  a  base’s  existing  infrastructure;  a  separate 
LOCIS  server  is  not  recommended.  Participants  perceived  the  proximity  device  as  an  additional 
and  separate  logon — with  additional  password  requirements,  another  card  to  manage,  and  a 
security  risk  in  those  instances  when  it  is  lost.  As  such,  the  group  was  not  receptive  to  including 
it  as  a  LOCIS  capability.  The  group  did  acknowledge  that  the  concept  of  a  common  access  badge 
is  likely  to  be  a  forthcoming  capability  across  the  Air  Force.  Another  issue  of  concern  to  SLAP 
participants  was  the  data  driving  LOCIS,  namely,  sources  of  data  and  how  LOCIS  would 
accommodate  for  instances  in  which  data  become  inaccessible. 

B.l.2.1  Data  Management 

In  considering  data  management  aspects  of  LOCIS,  participants  provided  comments  on  the 
snapshot/book  marking  features  and  the  pivot  table  capability. 

Snapshot  and  Book  Marking  Features 

SLAP  members  were  receptive  to  the  inclusion  of  both  snapshot  and  book  marking  features, 
although  not  all  participants  were  in  favor  of  using  the  book  marking  feature  (providing  links  to 
real-time  data)  to  support  stand-up  meetings.  While  they  recognized  the  importance  of 
maintaining  ready  access  to  real-time  data  (e.g.,  financial  and  wholesale/contractors  databases), 
some  participants  indicated  they  would  not  be  interested  in  using  it  during  stand-up  briefings 
because  in  most  cases,  they  would  have  no  opportunity  to  review  and  analyze  real-time  data 
immediately  prior  to  a  meeting.  They  suggested  they  would  rather  discuss  data  with  which  they 
were  familiar  and  had  previously  reviewed — and  as  such,  would  prefer  to  use  the  PowerPoint 
snapshot  feature  in  preparing  for  stand-up  meetings.  Including  a  time  stamp  on  PowerPoint 
snapshots  was  suggested.  In  discussing  the  benefits  of  the  PowerPoint  snapshot  feature  in 
supporting  stand-up  meetings,  participants  acknowledged  its  ability  to  offer  a  manpower  savings 
with  respect  to  preparation  time. 

Pivot  Table 

In  responding  to  questions  addressing  (1)  the  extent  to  which  an  LG/OG  might  use  a  pivot  table 
to  manipulate  data  and  (2)  the  types  of  tasks  that  might  be  supported  by  such  a  tool,  feedback 
was  mixed.  The  differences  across  the  group  as  a  whole  suggested  use  of  the  pivot  table  would 
depend  upon  personal  preferences  and  management  styles  of  individual  users.  Some  members 
expressed  an  interest  in  applying  it  to  their  work,  while  others  did  not.  Ultimately,  participants 
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recommended  it  remain  available  as  a  user  option.  Its  potential  usefulness  to  DOG/MA  users,  in 
particular,  for  supporting  trend  analyses,  was  also  suggested. 

From  reliability,  validity,  and  accuracy  perspectives,  the  source(s)  of  pivot  table  data  was  an 
issue  of  concern.  Information  from  other  bases,  for  example,  may  be  suspect.  Assessing  the 
pros/cons  of  CAMS  versus  REMIS  data  will  likely  be  an  issue. 

B.l.2.2  Dynamic  User  Interface 

In  considering  aspects  of  the  LOCIS  dynamic  user  interface,  participants  commented  on  the 
critical  information  elements  assigned  to  OG  and  LG  default  workspaces,  as  well  as  the 
appropriateness  of  the  wing  capability  indicator.  Some  participants  also  expressed  a 
receptiveness  to  the  concept  of  customizing  information  displays  to  accommodate  individual 
preferences. 

Integrated  Flying  Schedule  and  Aircraft  Status  Information 

Some  participants  suggested  that  both  OG  and  LG  default  workspaces  contained  too  much  detail 
for  the  types  of  activities  required  at  OG/LG  levels.  Such  detail  might  be  more  appropriate  for 
DOG/MA  or  perhaps  Senior  Management  Office  (SMO)  activities.  One  concern  was  that  too 
much  detail  “opens  the  door  to  micromanagement.” 

Members  suggested  additional  data  for  OG  and  LG  default  workspaces.  Additional  data 
suggestions  for  the  OG  default  workspace  are  as  follows: 

a.  Aircrew  availability 

b.  MC  status 

c.  Refueling  status/air  refueling  status 

d.  Training 

e.  Munitions  loading 

f.  Range  times 

g.  Information  from  other  bases 

h.  Air  field  status 

i.  SOF  status 

Suggestions  for  the  LG  default  workspace  are  as  follows: 

a.  Equipment  status 

b.  Spare  engines 

c.  Munitions 

d.  Pallets — indicating  mobilization  capability 

e.  Multiple  MDS. 

Some  participants  felt  at  the  LG  level,  the  aircraft  configuration  information  provided  via 
drill-down  from  aircraft  status  icons  was  unnecessary. 
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Wing  Capability  Indicator 


SLAP  members  were  skeptical  about  the  potential  usefulness  and  appropriateness  of  a  wing 
capability  indicator.  From  their  perspective,  it  satisfied  no  need,  and  they  were  not  receptive  to 
its  inclusion  on  the  LOCIS  web  page.  Participants  felt  that  a  parameter  such  as  capability  could 
not  be  defined  or  measured  in  a  meaningful  manner.  From  their  perspective,  a  snapshot 
measurement,  as  offered  by  the  gauge  depicted  on  the  web  page,  does  not  adequately  support  the 
types  of  decisions  typically  reached  by  LG  and  OG  users.  Because  of  its  limited  power  in 
supporting  decision-making  activities,  participants  were  skeptical  of  its  usefulness.  From  their 
perspective,  a  more  useful  indicator  would  be  one  assessing  how  well  the  wing  is  meeting 
required  plans — or  one  reflecting  contingency  response  time. 

SLAP  comments  suggested  that  a  single  indicator  with  no  drill-down  capability  would  be 
unusable.  However,  in  response  to  proposed  drill-down  data  depicting  status  of  resource 
availability,  where  resources  included  Maintenance,  Munitions,  Supply,  Fuels,  Transportation, 
and  Operations,  SLAP  members  indicated  the  term  availability  had  limited  meaning.  The 
underlying  concern  is  that  terms  such  as  capability  and  availability  are  subjective.  At  a 
minimum,  any  measurement  of  wing  capability  must  be  customizable,  where  the  parameters 
driving  wing  capability  are  user  definable.  Arriving  at  a  universal  definition  of  wing  capability 
would  be  impossible.  Members  also  indicated  that  the  data  of  potential  use  in  measuring  wing- 
level  capability  is  likely  to  be  classified.  Other  feedback  indicated  mobile  readiness  data  is 
currently  available  through  AF  smart  cards. 

B.l.2.3  Proactive  Decision  Support 

In  considering  aspects  of  proactive  decision  support  provided  through  LOCIS,  participants 
commented  on  alerts  and  warnings  messages,  user-definable  thresholds,  and  the  “what-if’ 
simulation  capability. 

Alerts  and  Warnings 

In  discussing  how  alert  or  warning  messages  should  be  managed  once  the  emergency  conditions 
triggering  them  are  resolved,  participants’  perspectives  varied.  In  some  instances,  participants 
expressed  a  preference  for  a  “hands-on”  management  style  in  which  all  messages  would  be 
displayed  until  manually  saved  or  deleted.  On  the  other  hand,  preferences  for  less  hands-on 
control  were  also  expressed,  where  messages  would  be  automatically  deleted  once  they  had  been 
reviewed. 

In  considering  options  for  maintaining  a  log  of  all  messages,  one  suggestion  was  for  LOCIS  to 
provide  an  auto-archiving  capability,  where  alerts  and  warnings  might  be  logged  according  to 
date  and  MDS. 


40 


User-Definable  Thresholds 

SLAP  members  want  the  option  of  selecting  the  thresholds  of  specific  importance  to  them, 
although  they  recognized  that  adherence  to  command  settings  might  be  an  appropriate  use  of  the 
capability.  Participants  also  identified  “holes”  in  the  list  of  thresholds  displayed  in  the  Threshold 
Settings  view.  Thresholds  for  monitoring  conditions  specific  to  the  following  resources  were 
suggested: 

a.  Critical  Auxiliary  Ground  Equipment  (AGE) 

b.  Personnel/manpower 

c.  Supply 

d.  MHE  (cargo  loading) 

While  recognizing  the  importance  of  personnel  and  supply  thresholds,  participants  acknowledged 
the  difficulty  in  defining  them.  (From  what  source(s)  is  manpower  availability  coming?  For 
supply  data,  how  are  unit  priorities  (Unit  X  versus  Unit  Y  with  respect  to  AEG/Contingency 
Operations)  considered?)  SLAP  members  also  raised  questions  about  the  appropriateness  of 
thresholds  currently  listed.  LG/OG  users,  for  example,  would  not  wait  for  LOCIS  to  provide  IFE 
alerts. 

In  response  to  questions  about  the  usefulness  of  the  capability,  one  suggestion  was  to  set  up  a  test 
bed  among  potential  users,  providing  them  with  a  set  of  adjustable  thresholds  to  use  and  evaluate 
in  a  realistic  environment. 

What-If  Simulation 

Comments  on  the  applicability  of  “what-if’  simulation  analyses  were  mixed,  reflecting 
differences  in  user  preferences  and  management  styles.  Most  participants  viewed  it  as  a  useful 
capability,  but  the  differences  in  comments  suggested  its  use  would  be  dependent  upon 
individual  user  style.  In  considering  sources  of  the  data  used  as  inputs  to  simulation  analyses, 
one  suggestion  was  to  incorporate  a  capability  to  share  data  with  other  bases.  The  impact  of  this 
particular  simulation  capability  on  the  use  of  the  Dyna-METRIC  Microcomputer  Analysis 
System  (DMAS)  or  on  other  forecasting  tools  (i.e.,  how  it  might  interface  with  or  replace 
existing  tools)  was  of  interest. 

B.l.2.4  Real-Time  Information  Updating 

SLAP  members  felt  that  notifications  of  real  emergencies  would  likely  come  via  radio  and/or 
phones  and  that  the  Alerts  and  Warnings  window  would  not  be  the  primary  means  through  which 
users  are  notified  of  real  emergencies.  LOCIS  users  would  be  notified  at  the  same  time  the  MOC 
is  notified. 

Voice-Bused  S tutus  Changes 

SLAP  members  recognized  a  need  for  some  type  of  feedback  mechanism  in  which  the  MOC 
would  acknowledge  and  verify  status  changes.  One  recommendation  was  to  enable  this 
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capability  to  work  with  the  back  shops  to  include  job  status/job  completion,  backorder  parts, 
links  to  SBSS,  and  links  to  flying/maintenance  schedules. 

Notification  of  Alert  and  Warning  Messages 

Cell  phone  vulnerabilities  to  jamming  and  adverse  weather  conditions,  thunderstorms  in 
particular,  were  perceived  as  a  risk  among  some  members.  In  one  instance,  the  notification  of 
IFE  and  ground  emergencies  over  a  cell  phone  was  viewed  as  unrealistic.  Adding  a  cell  phone  to 
the  LG/OG  collection  of  portable  devices  was  a  concern.  In  reacting  to  the  MICAP  warning, 
some  members  identified  a  need  to  provide  a  link  to  real-time  supply  data  (SBSS,  regional  supply 
information).  In  considering  the  manner  in  which  users  are  notified  of  incoming  LOCIS-specific 
e-mail  messages,  one  recommendation  was  that  such  messages  be  highlighted  to  cue  users  of 
their  arrival. 

B.1.3  LOCIS  User  Group  Data 

As  with  the  SLAP  data,  inputs  from  members  of  the  LUG  were  characterized  according  to  one  of 
the  four  LOCIS  capabilities  highlighted  during  the  Year  1  demonstration:  data  management, 
dynamic  user  interface,  proactive  decision  support,  and  real-time  updating  of  information.  Each 
of  these  aspects  was  assessed  through  a  series  of  questions,  where  die  questions  in  a  given  series 
were  designed  to  focus  LUG  participants’  comments  on  a  set  of  supporting  features.  In  addition 
to  the  verbal  feedback  offered  during  this  question-and-answer  session,  participants  also,  in  some 
instances,  provided  written  comments  to  the  questions.  During  the  LOCIS  Year  1 
Demonstration,  each  feature  set  was  implemented  to  demonstrate  how  its  respective  capability 
might  be  supported  within  LOCIS.  For  each  capability,  a  summary  statement  is  provided. 
Following  each  summary  statement,  further  details  specific  to  the  supporting  features  are 
documented.  Note  that  verbal  feedback  obtained  during  the  question-and-answer  session,  as  well 
as  written  comments,  is  documented  in  this  section. 

B.l.3.1  Data  Management 


Summary 

The  data  driving  LOCIS  was  of  interest  to  LUG  members,  specifically,  sources  of  data,  how 
LOCIS  would  operate  when  respective  databases  are  inaccessible,  and  the  timeliness  of  LOCIS 
data.  One  identified  risk  is  LOCIS’  reliance  on  systems  external  to  a  base’s  firewall,  and  a 
recommendation  to  notify  LOCIS  users  when  such  systems  are  inaccessible  or  unavailable  was 
offered.  Incorporating  a  time  stamp  on  LOCIS  data  screens,  as  well  as  providing  built-in 
refreshes,  were  suggested  as  methods  for  enhancing  user  situation  awareness  from  a  perspective 
of  data  timeliness. 

LOCIS  compatibility  with  legacy  systems  was  also  a  concern  for  users.  Incompatibility  with  a 
given  legacy  system  that  currently  receives  extensive  use  may  limit  or  preclude  continued  use  of 
that  system  once  use  of  LOCIS  is  adopted.  LUG  participants  recommended  that  during  the 
Year  2  effort,  the  LOCIS  team  identify  compatibilities,  as  well  as  incompatibilities,  with  legacy 
systems. 
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In  discussing  accessibility  to  detailed  information  through  a  drill-down  capability,  LUG 
members  validated  the  current  underlying  design  strategy  in  which  the  critical  data  elements 
defining  a  given  information  display  are  identified  and  presented  at  the  top  level  of  the 
display,  while  secondary  or  supporting  data  are  contained  in  a  lower  layer.  Specifically,  they 
stated  LOCIS  should  provide  answers  to  critical  questions  (e.g.,  why  a  line  number  is  not 
ready)  at  the  top  level  of  a  given  display,  not  through  its  drill-down  capability.  In  other 
words,  users  should  not  be  forced  to  obtain  critical  data  through  a  series  of  clicks  or  double 
clicks  on  numerous  active  regions  of  a  display. 

Snapshot  and  Book  Marking  Features 

LUG  participants  anticipated  LG/OG  receptiveness  to  the  capability  allowing  them  to  capture 
and  save  snapshots  of  data  as  supported  by  LOCIS’  PowerPoint  capture  feature.  Although  some 
bases  currently  support  stand-up  meetings  with  real-time  data  (e.g.,  Eglin),  the  expectation  is  that 
not  all  LG/OG  users  will  be  receptive  to  using  LOCIS’  bookmarking  feature  (providing  links  to 
real-time  data)  to  support  of  stand-up  meetings. 

Pivot  Table 

Users  viewed  the  data  manipulation  capability  afforded  by  the  pivot  table  to  be  useful  and 
suggested  that  repeat/recur  data  would  be  a  good  use  of  pivot  table  capability.  Titles/labels  for 
graphs  generated  “on  the  fly”  from  a  pivot  table  must  reflect  the  data  selected  in  the  sort 
operation  and  subsequently  displayed  in  the  graphs.  During  the  demonstration,  graphs  generated 
from  new  sorting  operations  were  not  automatically  relabeled.  Rather,  the  label  for  the  original 
graph  (i.e.,  the  graph  displayed  prior  to  the  sorting  operation)  was  displayed  with  the  new  graph, 
making  it  inconsistent  with  the  data  actually  depicted.  Users  indicated  that  this  inconsistency 
was  unacceptable  and  that  all  titles/labels  should  provide  a  meaningful  description  of  the  data 
depicted.  Users  also  identified  a  need  to  compare  data  across  a  subset  of  the  bases  listed  and 
acknowledged  the  pivot  table’s  usefulness  in  supporting  a  sort  of  that  nature. 

Drill-Down  Capability 

For  LOCIS  graphics-based  data  displays  (e.g.,  aircraft  status  icons,  bar  charts  depicting  recap 
data,  flying/maintenance  schedule  time  lines  depicting  “scheduled”  versus  “actual”  activities), 
users  prefer  that  a  drill-down  capability  be  provided  through  direct  interaction  with  the  graphic 
itself— i.e.,  through  selection  of  a  “hot  spot”  (link)  on  the  graphic — as  demonstrated  for  the 
aircraft  status  icons.  Users  identified  other  LOCIS  graphics-based  data  displays  (bar  charts,  in 
particular)  as  appropriate  for  this  type  of  drill-down  activity.  Specifically,  each  bar, 
corresponding  to  one  of  the  categories  summarized  in  the  bar  chart  (e.g.,  On-Time  Takeoffs,  MC 
Rate,  UTE  Rate  displayed  in  Yesterday’s  Wing  Recap  graph),  would  be  active.  Users  could  click 
on  the  bar  corresponding  to  category  MC  Rate,  for  example,  to  obtain  further  information  about 
the  data  driving  the  rate  summarized  at  the  top  level  of  Yesterday’s  Wing  Recap  graph. 
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B.l.3.2  Dynamic  User  Interface 
Summary 

LUG  participants  were  receptive  to  the  concept  of  a  customizable  user  interface  in  which  a  range 
of  “set-up”  options  allows  them  to  satisfy  individual  preferences  in  selecting  the  data  elements 
actually  displayed  (e.g.,  bar  chart  categories),  specifying  how  data  should  be  displayed  (e.g., 
graphically  versus  textually),  and  determining  how  data  should  be  sorted  (e.g.,  user-specified 
time  periods  for  recaps  and  histories— daily  versus  weekly  versus  monthly).  At  the  same  time, 
LUG  members  emphasized  the  importance  of  designing  an  appropriate  starting  point  (default)  for 
content,  format,  and  control  of  workspaces.  “General  Jumper’s  message,”  for  example,  might 
provide  the  default  for  the  Recap. 

Participants  found  the  graphical  depiction  of  flying  schedule  and  aircraft  status  data  (i.e.,  a  time 
line  distinguishing  between  scheduled  and  actual  flying/maintenance  activities)  to  be  a  useful 
display  technique.  They  also  identified  the  integration  of  flying  schedule  data  and  maintenance 
schedule  data  as  useful. 

LUG  members  emphasized  the  importance  of  designing  multiple-view  workspaces  such  that  the 
relationships  linking  data  between  two  or  more  views  are  explicit. 

LUG  members  anticipated  that  current  LOCIS  displays  would  likely  provide  too  much  detail  for 
LG  and  OG  users.  They  identified  the  Threshold  Settings  view  as  one  more  likely  to  be  used  by 
lower  level  maintenance  managers.  Along  these  same  lines,  LUG  members  suggested  that 
LG/OG  users  will  most  likely  be  interested  in  first  reviewing  a  top-level  graphical  overview  (in 
which  “red”  versus  “green”  aircraft  are  identified  as  such)  and  then  drilling  down  from  the 
overview  information  to  obtain  further  details.  Another  recommendation  was  to  include  a 
summary  capability  within  the  flying  schedule  and  aircraft  status  displays  as  a  means  of  depicting 
an  at-a-glance  overview  of  aircraft  status. 

LUG  members  proposed  several  regions,  objects,  and  data  elements  on  the  LOCIS  flying 
schedule  and  aircraft  status  displays  as  potential  links  to  drill-down  data.  Identified  as  candidate 
active  “hot  spots”  were  line  numbers  and  scheduled  and  actual  segments  of  the  flying  schedule 
and  maintenance  schedule  time  lines.  Also  identified  were  other  types  of  drill-down  data 
currently  unavailable  in  LOCIS  displays. 


With  respect  to  incorporation  of  the  proximity  device  as  a  role-based  login/logout  mechanism, 
LUG  members  identified  security  and  deployability  (different  requirements  for  RF  overseas)  as 
two  issues  for  further  consideration. 

User  Preferences 

LUG  participants  identified  a  number  of  parameters  that  might  be  included  among  the  set-up 
options  available  for  user  customization.  One  suggestion  was  to  allow  users  to  Alter  data  items 


44 


for  display  via  selection  (and  de-selection)  from  a  checklist  of  items.  They  also  suggested  a 
range  of  areas  for  customizing  data  displays: 

a.  Allow  data  categories  (e.g.,  recap  categories)  to  be  user  selectable.  Identify  an 
appropriate  set  of  default  categories  and  allow  users  to  add  others. 

b.  Allow  the  data  provided  as  part  of  a  drill-down  (e.g.,  tail  number,  location,  configuration 
code,  line  number  data  displayed  via  drill-down  from  aircraft  status  icons)  be  user 
selectable. 

c.  Allow  time  periods  over  which  data  are  summarized  (e.g.,  recaps,  histories)  to  be  user 

selectable  up  to  30  days — i.e.,  recap  as  of  time  period: _ ,  7-Day  History  versus  30- 

Day  History. 

d.  Allow  the  “clock”  represented  on  the  flying  and  maintenance  schedule  time  lines  to  be 
user  selectable  (24-hour  versus  zulu  versus  local  time).  This  option  would  be  useful 
during  deployments. 

Integrated  Flying  Schedule  and  Aircraft  Status  Information 

LUG  members  would  prefer  to  see  a  single  tail  number  entry  on  flying  schedule  and  aircraft 
status  displays,  rather  than  a  tail  number  entry  for  each  successive  run.  Their  preference  is  for 
multiple  runs  to  be  depicted  on  a  single  time  line.  Participants  suggested  that  upon  detection  of  a 
potential  problem  in  meeting  the  next  run,  LOCIS  might  provide  a  cue  (e.g.,  color  code  the  next 
scheduled  activity  in  red).  Additionally,  to  support  rapid  responses  during  contingencies,  users 
emphasized  the  need  for  ready  identification  of  spare  aircraft,  as  well  as  open  line  numbers.  An 
OG  would  be  most  interested  in  knowing  whether  a  sufficient  number  of  aircraft  are  available  to 
fill  the  flying  schedule.  Participants  suggested  that  linking  maintenance  manpower  data  to  the 
LG  default  information,  including  data  on  phase  capabilities,  skill  levels,  workload  capability, 
work  center  workload  (percent  level  at  which  a  unit  is  working),  would  be  appropriate. 
Information  on  engine  status  would  also  be  helpful. 

While  accurate  aircraft  status  information  is  necessary  in  supporting  decision-making  activities, 
LUG  participants  indicated  it  is  not  sufficient.  Line  status,  as  well  as  the  status  of  blocks  of  lines 
(i.e.,  status  according  to  the  generation  sequence)  is  also  critical.  An  aircraft  may  have  an  FMC 
status,  but  it  may  not  be  “green”  in  its  sequence  (e.g.,  cargo  is  late  and  has  not  yet  arrived,  crew  is 
late).  Thus,  LOCIS  should  provide  status  with  respect  to  generation  sequence  (e.g.,  weapons 
load  status,  fuel  status,  cargo  load  status).  This  status  might  be  referred  to  as  mission  status 
readiness.  Participants  suggested  that  such  status  information  might  be  incorporated  as  drill¬ 
down  data  accessible  from  the  line  number  (via  a  cursor  roll-over  or  mouse  click).  Another 
option  might  be  to  display  icons  along  the  flying  schedule  time  line,  each  one  representing  a 
stage  in  the  generation  sequence.  Icon  appearance  would  change  to  reflect  status  of  individual 
stages  (ready  versus  not  ready)  in  the  sequence.  Other  information  available  via  a  cursor  roll¬ 
over  might  be  WUC.  Participants  indicated  the  need  to  identify  “on  alert”  aircraft,  as 
notifications  regarding  the  launching  of  such  aircraft  are  sent. 
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LUG  participants  suggested  a  new  “tab”  for  the  OG  Default  Workspace:  Flying  Schedule  -  PMC 
Only.  LUG  participants  also  suggested  that  aircraft  status  icons  remain  fixed  as  the  flying 
schedule  and  aircraft  status  displays  are  scrolled  horizontally. 

Drill-Down  Data 

LUG  participants  proposed  a  set  of  active  “hot  spots”  and  corresponding  drill-down  data.  Their 
suggestions  are  summarized  in  Table  B.2.  As  indicated  in  Table  B.2,  participants  provided  a 
number  of  observations  and  recommendations  addressing  configuration  information,  among 
them  a  suggestion  for  more  generic  configuration  codes  (e.g.,  air-to-air,  air-to-ground)  than  those 
currently  displayed  upon  drill-down  from  the  aircraft  status  icons — with  more  detailed  codes 
available  via  another  layer  of  drill-down.  They  observed  that  a  capability  allowing  users  to 
compare  an  aircraft’s  current  (actual)  configuration  with  its  required  (scheduled)  configuration 
would  be  useful,  suggesting  a  drill-down  capability  from  line  and  tail  numbers  to  obtain  required 
and  current  configuration,  respectively.  Participants  observed,  however,  that  receiving  a  cue 
when  a  mismatch  between  required  and  current  configurations  exists  might  be  more  useful. 

Another  type  of  drill-down  data  currently  unavailable  from  LOCIS  but  suggested  by  LUG 
members  was  a  list  of  parts  removed  from  a  cann  aircraft. 


Table  B.2.  LOCIS  Links  Proposed  by  LUG  Members 


Hot  Spot 

Drill-Down  Data 

Line  number 

•  Sortie  debrief/recap  information  (e.g.,  why  A/C  landed  before 

scheduled  landing  time) 

•  Line  status  (i.e.,  according  to  generation  sequence) 

•  Required  configuration 

Tail  number 

•  Current  configuration 

unscheduled  maintenance  time  periods  on 

•  Cause  of  unscheduled  activity 

Aircraft  Status  view 

•  MICAP  data 

scheduled  and  actual  time  periods  on 

•  Primary  problem  with  NMC  aircraft 

Aircraft  Status  view 

scheduled  time  periods  on  Flying  Schedule 

•  Missing  items  (e.g.,  fuels,  bombs) 

•  Required  configuration 

actual  time  periods  on  Flying  Schedule 

•  Current  configuration 

aircraft  status  icons 

•  Items  missing  from  required  configuration 

MICAP  part  listed  in  MICAP  view 

•  Maintenance  schedule 

Multiple-View  Workspaces 


LUG  participants  were  receptive  to  LOCIS’  multiple- view  workspaces  but  emphasized  the 
importance  of  designing  information  displays  such  that  the  relationships  linking  data  elements  of 
two  or  more  views  are  readily  apparent.  For  LUG  participants,  the  need  for  displays  in  which 
noteworthy  data  relationships  are  easily  recognized  was  of  particular  interest  in  the  tri-pane 
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workspaces  designed  for  alert  and  warning  information.  LUG  members  suggested  that  LOCIS 
clearly  identify  the  “drivers”  of  a  given  result  or  impact  (e.g.,  MICAP  report,  7-day  IFE 
history) — i.e.,  link  the  result/impact  to  the  data  driving  it.  Specific  LUG  suggestions  for 
enhancing  the  visual  depiction  of  data  links  are  described  in  the  section  addressing  proactive 
decision  support. 

Participants  also  indicated  an  interest  in  reviewing  different  “levels”  of  MICAP  data  (e.g.,  wing- 
and  squadron-level  MICAPs,  as  well  as  MICAPs  for  a  given  aircraft). 

B.l.3.3  Proactive  Decision  Support 


Summary 

LUG  participants  supported  LOCIS’  proactive  decision  support  capabilities  and  felt  those 
demonstrated  during  Year  1— namely,  alerts  and  warnings,  user-definable  thresholds,  and  the 
“what-if”  simulation  tool — to  be  useful  components.  They  also  anticipated  that  LG/OG  use  of 
the  threshold  adjustment/selection  capability  and  the  simulation  tool  would  be  based  on  the 
individual  preferences  and  management  styles  of  an  LG  or  OG. 

Alerts  and  Warnings 

LUG  members  were  satisfied  with  the  interaction  approach  required  for  responding  to  alerts  and 
warnings  received  via  the  Alerts  and  Warnings  window.  Upon  login,  they  would  prefer  not  to 
receive  alert  or  warning  messages  for  emergencies  already  resolved  and  suggested  they  would  not 
be  interested  in  manually  identifying,  sorting,  and  deleting  messages  that  no  longer  reflected 
actual  emergency  conditions.  LUG  members  also  expressed  a  concern  that  mailboxes  would  fill 
up  quickly  and  suggested  automatic  deletion  of  those  messages  not  read  within  a  designated 
period  of  time.  Participants  did  not  perceive  the  assignment  of  a  level  of  criticality  or  severity  to 
a  given  alert  or  warning  (e.g.,  high,  medium,  low)  as  a  need. 

User-Definable  Thresholds 

LUG  members  acknowledged  the  usefulness  of  the  ability  to  select  thresholds  from  a  predefined 
list  and  to  adjust  alert  and  warning  levels  for  each  threshold.  They,  suggested  that  analysis 
personnel  might  establish  initial  threshold  settings.  They  also  identified  a  need  to  create  new 
thresholds,  a  capability  under  consideration  for  future  demonstration.  Specifically,  creating  a 
new  threshold  might  arise  from  the  need  to  monitor  unique  (non-routine)  conditions  (e.g.,  a 
one-time  inspection).  To  support  the  monitoring  of  atypical  conditions,  “special  purpose”  or 
“one-time  use”  thresholds  would  be  created.  In  creating  new  thresholds,  participants  suggested 
an  approach  allowing  them  to  select,  edit,  or  define  “key  events”  (perhaps  via  a  list  of  key 
words — e.g.,  WUC),  where  occurrences  of  key  events  would  trigger  alerts  or  warnings.  For  most 
events,  linking  to  a  WUC  is  possible,  but  isolating  according  to  MDS  would  also  be  required. 
While  participants  rated  the  user-definable  threshold  capability  as  highly  useful,  they  suggested 
extensive  use  ot  the  capability  would  most  likely  be  among  lower  level  management  roles. 

When  asked  to  critique  the  threshold  categorization  approach  implied  within  the  Threshold 
Settings  view,  LUG  participants  found  it  suitable.  Another  suggestion  was  to  follow  a 
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“functional”  grouping,  but  the  participants’  definition  of  the  term  functional  was  not  explored. 
One  interpretation  might  be  functional  groups  that  include  Supply,  Transportation,  POL, 
Personnel,  and  Facilities.  They  also  suggested  inclusion  of  a  Reset  All  feature  that  would  reset  all 
thresholds  to  their  default  settings  at  once.  Several  “holes”  in  the  list  of  thresholds  displayed  in 
Threshold  Settings  were  identified,  and  LUG  members  acknowledged  the  current  list  of 
thresholds  is  not  comprehensive  (i.e.,  it  does  not  reflect  all  conditions  potentially  monitored). 
Examples  of  thresholds/scenarios  missing  from  the  list  are  as  follows: 

a.  Ground  abort 

b.  Ground  emergency 

c.  EOR  situations 

d.  Attack  warnings  (generic  across  MAJCOMs) 

e.  Launch  of  an  alert  aircraft 

f.  Situations  arising  during  Phase  2  scenarios  (alarm  red/black,  FOD  level),  as  well  as  those 
of  interest  from  a  maintenance  perspective  during  Phase  2  scenarios  (contaminated  A/C, 
A/C  recovering  from  an  attack,  A/C  damaged  in  a  dogfight) 

Another  suggested  feature  was  one  allowing  users  to  view  management’s  threshold  settings. 

Displaying  data  relationships  such  that  they  are  readily  identified  was  of  particular  interest  in  the 
tri-pane  displays  designed  for  alert  and  warning  workspaces.  LUG  members  suggested  that 
LOCIS  clearly  identify  the  “drivers”  of  a  given  result  or  impact  (e.g.,  MICAP  report,  7-day  IFE 
history)— i.e.,  link  the  result/impact  to  the  data  driving  it.  In  the  IFE  Alert  workspace,  for 
example,  if  a  subset  of  the  items  listed  in  the  7-Day  History  view  is  driving  a  result  depicted  in 
the  IFE  summary  bar  chart,  LOCIS  might  highlight  that  subset  of  items  upon  user  selection  of  the 
respective  bar  chart  category.  In  the  MICAP  Warning  workspace,  users  suggested  enhancements 
to  the  manner  in  which  relationships  between  the  MICAP  Warning,  MICAP,  and  Aircraft  Status 

-  NMC  Only  views  are  depicted.  Specifically,  users  found  the  MICAP  Warning  view 
unnecessary.  They  also  suggested  the  MICAP  view  include  fields  for  source  of  supply  and  mode 
of  transportation  and  that  the  field  for  NSN  be  removed.  LUG  members  recommended  that  any 
link  between  parts  listed  in  the  MICAP  view  and  unscheduled  maintenance  activities  appearing 
in  the  Aircraft  Status  -  NMC  Only  view  be  identified.  One  approach  might  be  to  highlight  the 
MICAP  data  (MICAP  view)  corresponding  to  an  unscheduled  maintenance  event  (Aircraft  Status 

-  NMC  Only  view)  upon  user  selection  of  the  segment  of  the  time  line  associated  with  that  event. 

What-If Simulation 

While  LUG  members  acknowledged  the  usefulness  of  LOCIS’  “what-if’  simulation  capability, 
they  viewed  it  to  be  more  of  a  “lower  level”  management  tool.  Typically,  action  items  are 
generated  from  lower  level  supervisors  (namely,  squadron  commanders,  maintenance  officers,  or 
production  superintendents)  and  reported  to  higher-level  management.  In  this  manner, 
simulation  results,  from  which  lower  level  supervisors  might  derive  a  set  of  action  items,  would 
be  reported  up  to  the  LG/OG  level.  Although  participants  anticipated  LG/OG  use  of  the 
simulation  capability  to  be  driven  primarily  by  individual  preferences  and  management  style, 
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they  did  suggest  its  potential  usefulness  to  OG  commanders  and  wing  schedulers  in  making  out- 
year  predictions  (capabilities  projections)  across  a  broad  range  of  wing  scheduling  activities, 
including  next  year’s  flying  hour  program  (answering  questions  on  the  number  of  sorties  a  wing 
can  produce  with  a  given  level  of  manpower  and  facilities),  training  requirements,  and  higher 
headquarters  directives. 

Participants  suggested  that  the  manner  in  which  LOCIS  currently  presents  simulation  results 
would  be  enhanced  if  projections  were  displayed  in  conjunction  with  some  baseline.  In  this 
manner,  the  impact  of  a  set  of  anticipated  conditions  is  more  readily  assessed  when  it  is 
compared  against  results  corresponding  to  a  set  of  baseline  conditions. 

B.l.3.4  Real-Time  Information  Updating 


Summary 

LUG  participants  viewed  data  accuracy  and  timeliness  as  critical  to  the  success  of  logistics, 
maintenance,  and  operations  activities.  They  suggested  potential  LOCIS  users  would  be  “loyal” 
to  their  radios  and  would  typically  view  them  as  their  most  important  mobile  communications 
device.  Currently,  radios  are  the  primary  notification  device.  Participants  suggested  potential 
users  would  not  be  waiting  for  the  LOCIS  Alerts  and  Warnings  window  to  notify  them  of 
emergency  situations.  While  cell  phone  use  is  not  uncommon  among  LG  and  OG  users, 
participants  cautioned  that  many  would  not  be  enthusiastic  about  a  requirement  to  carry  another 
device.  A  new  technology  recently  introduced  by  Motorola  in  which  radio  and  cell  phone 
capabilities  are  combined  in  a  single  mobile  device  was  suggested  for  future  consideration. 

In  assessing  the  use  of  cell  phones,  participants  identified  battery  life  as  an  issue  to  consider. 
Security,  deployability,  and  vulnerability  of  cell  phone  and  RF  communication  capabilities  were 
identified  as  issues  to  be  addressed  in  the  coming  year.  Jamming  was  identified  as  a  critical 
vulnerability  of  cell  phones  and  RF  capabilities. 

Voice-Based  Status  Changes 

LUG  participants  were  receptive  to  the  LOCIS  capability  allowing  voice-based  status  updates. 
At  the  same  time,  they  recognized  a  need  to  consider  the  checks  and  balances  typically  applied 
whenever  aircraft  status  is  changed  (e.g.,  only  the  MOC  has  authority  to  change  aircraft  status, 
yet  Expediters  and  Production  Superintendents  are  often  accountable  for  all  data).  LOCIS  must 
accommodate  the  policy  rules  in  effect  at  a  given  base.  Participants  recommended  the 
incorporation  of  some  type  of  feedback  mechanism  in  which  the  MOC  acknowledges  and 
verifies  status  changes.  They  also  recognized  that  this  type  of  updating  capability  does  not 
necessarily  reduce  a  vulnerability  to  erroneous  data.  This  issue  of  data  quality  will  become 
increasingly  important  upon  integration  with  CAMS. 

Other  status-type  information  that  might  be  updated  include  aircraft  configuration  (weapons 
loading,  fuel),  last  fly  dates,  number  of  days  down,  hangar  queen  status,  POL,  and  munitions. 
Handling  tail  swaps  is  also  an  issue  suggested  for  further  consideration. 

Notification  of  Alert  and  Warning  Messages 
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When  asked  how  LOCIS-specific  (i.e.,  role-specific)  e-mail  messages  should  be  managed,  LUG 
participants  suggested  that  such  e-mail  messages  be  separated  from  alert  and  warning  messages. 

Two  mailboxes— one  for  role-specific  e-mail  messages  and  one  for  alert/waming  messages _ 

were  suggested.  Routing  role-specific  e-mail  messages  to  a  user’s  e-mail  account,  instead  of  a 
LOCIS  account,  was  also  suggested.  In  responding  to  questions  about  the  order  in  which 
messages  should  be  listed,  participants  expressed  a  preference  for  a  mailbox  in  which  the  most 
recent  messages  are  displayed  at  the  top  of  the  list. 

In  accordance  with  the  LOCIS  approach  to  distinguish  between  “alert”  and  “warning”  conditions, 
participants  agreed  that  LOCIS  should  make  a  distinction  between  alerts  and  warnings.  Users 
indicated  a  need  to  be  cued  upon  arrival  of  incoming  messages  and  recommended  that  LOCIS 
provide  some  type  of  cue  (visual  or  auditory)  to  signal  the  arrival  of  a  new  message  in  the  Alerts 
and  Warnings  window. 

Participants  recommended  that  any  notification  parameters  for  alerts  and  warnings — e.g.,  cue 
type  (visual  versus  auditory  versus  both),  hardware  to  which  notifications  are  routed,  notification 
hours — be  specified  for  each  respective  threshold  on  an  individual  basis.  They  also  identified  the 
need  for  automatic  routing  of  notifications  to  a  default  device  during  the  periods  a  user  is  not 
logged  into  LOCIS. 

B.1.4  Lessons  Learned  and  Recommendations 

Based  on  the  data  collection  process  followed  during  the  LOCIS  Year  1  Demonstration,  we  have 
identified  several  lessons  learned  and  also  offer  recommendations  for  future  demonstration 
efforts. 

a.  Reduce  the  number  of  participants  in  the  LUG  data  collection  sessions.  Conducting  LUG 
sessions  with  fewer  individuals  is  recommended.  A  large  group  is  difficult  to  manage 
(i.e.,  too  many  concurrent  and  side  conversations  amongst  participants).  Under  these 
conditions,  maintaining  a  single  focus  is  difficult.  The  implication  for  Year  2  is  to 
conduct  more  LUG  data  collection  sessions  of  smaller-sized  groups  (preferably  no  more 
than  three  participants  per  question-and-answer  session)  similar  to  the  initial  LUG 
sessions  held  in  April  2000. 

b.  SLAP  members  were  very  willing  to  provide  feedback,  including  feedback  specific  to 
LOCIS  interface  design  and  usability.  Detailed  assessments  of  design  and  usability  are 
not  the  primary  data  collection  objectives  for  SLAP  usability  demonstrations;  at  the  same 
time,  providing  a  forum  in  which  SLAP  members  are  given  an  opportunity  to  critique 
design  and  usability  aspects  (perhaps  in  an  informal  discussion  group  format,  rather  than 
a  structured  question-and-answer  session)  is  recommended. 

c.  To  enhance  the  credibility  and  validity  of  our  user  data  from  a  human  factors  perspective, 
future  data  collection  efforts  should  provide  a  forum  in  which  users  have  an  opportunity 
to  interact  directly  with  LOCIS,  even  if  the  interaction  scenario  is  constrained  in  some 
manner. 
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B.2  Spiral  2 


During  Spiral  2,  the  Human  Factors  team  supported  design  of  the  user  interface  for  Block 
Release  and  Spiral  2  Demonstration  versions  of  the  LOCIS  capability.  Led  by  GTRI,  the  Human 
Factors  team  also  included  individuals  from  the  Air  Force  Research  Laboratory  (AFRL/HESR) 
and  the  University  of  Dayton  Research  Institute  (UDRI).  On  issues  specific  to  operational 
requirements,  the  team  received  support  from  the  Operational  Architecture  IPT. 

As  a  follow-up  to  the  user  interface  design  task — and  as  one  means  of  assessing  usability  of  the 
LOCIS  user  interface — the  Human  Factors  team  conducted  a  series  of  evaluation  sessions 
focusing  on  Block  Release  features.  Three  Block  Release  evaluation  sessions  were  conducted  at 
Hurlburt  Field,  FL  between  January  and  March  2002.  During  these  sessions,  Hurlburt  users 
provided  feedback  on  features  included  in  the  LOCIS  Block  Release.  A  single  evaluation 
session,  conducted  during  the  Spiral  2  Demonstration  held  at  BAE  Systems  on  30  April  2002, 
addressed  additional  features  highlighted  during  the  demonstration  and  not  provided  in  the  Block 
Release.  During  this  evaluation  session,  members  of  the  LUG  provided  feedback. 

A  final  Block  Release  evaluation  period  was  held  in  mid-June  2002  at  Hurlburt  Field.  Inputs 
from  users  who  had  participated  in  at  least  two  of  the  three  Block  Release  evaluation  sessions 
were  collected  through  a  ratings-based  questionnaire.  The  primary  purpose  of  this  final  data 
collection  period  was  to  provide  more  experienced  users  an  opportunity  to  consider  their  six- 
month  experiences  with  LOCIS  and  assess  its  capability  as  a  whole.  A  secondary  objective  was 
to  collect  preliminary  data  related  to  the  set  of  exit  criteria  established  by  AFRL/HESR.  These 
exit  criteria  will  become  the  main  area  of  focus  for  data  collection  during  the  LOCIS  field  test  to 
be  conducted  upon  completion  of  Spiral  3. 

This  report  documents  findings  of  the  data  collection  sessions  conducted  at  Hurlburt  Field  and 
during  the  Spiral  2  Demonstration.  It  also  documents  results  obtained  during  the  final  evaluation 
period  and  users’  perspectives  with  respect  to  the  LOCIS  exit  criteria. 

B.2.1  Data  Collection 

B.2.1.1  Block  Release  Data  Collection  Approach 

Members  of  the  LOCIS  Operational  Architecture  IPT  conducted  a  series  of  Block  Release 
training  sessions  in  November  2001.  Potential  users  of  the  Block  Release  were  identified  and 
invited  to  participate  in  the  November  training  session.  After  receiving  training,  these  new 
LOCIS  users  were  encouraged  to  interact  with  LOCIS  on  a  regular  basis.  The  two-month  period 
between  training  and  the  initial  data  collection  session  in  January  2002  afforded  new  users 
additional  time  in  which  to  familiarize  themselves  with  all  of  the  features  provided  in  the  Block 
Release.  Three  Block  Release  evaluation  sessions  were  conducted  at  Hurlburt  Field.  The  initial 
evaluation  session  was  conducted  22-25  January  2002.  The  second  and  third  sessions  were 
conducted  19-22  February  2002  and  19-22  March  2002,  respectively.  Of  the  33  users  trained  on 
LOCIS  in  November,  17  participated  in  the  evaluations.  Additionally,  8  of  these  17  participants 
provided  feedback  during  two  or  more  evaluation  sessions,  and  6  of  these  17  individuals 
participated  in  all  three  sessions. 
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During  each  month’s  data  collection  effort,  members  of  the  Human  Factors  team  conducted 
several  small  group  sessions.  Members  of  the  Operational  Architecture  IPT  provided  functional 
support.  In  each  small  group  session,  one  to  four  individuals  were  interviewed  on  various 
“usability”  aspects  of  a  set  of  pre-selected  LOCIS  features.  These  aspects  included  accessibility 
of  LOCIS  data,  ease  of  use,  and  usefulness  of  LOCIS  as  both  an  information  provider  and 
decision  support  tool.  User  feedback  specific  to  each  feature  was  recorded.  Users  also  provided 
subjective  ratings  on  the  features  targeted  for  each  month’s  evaluation  session.  (Note  that  the 
number  of  participants  in  each  small  group  session  was  necessarily  dictated  by  the  availability 
(work  schedule)  of  each  participant.) 

In  considering  accessibility  of  LOCIS  data,  users  provided  their  perceptions  on  the  accessibility 
of  the  LOCIS  web  site  (i.e.,  log  on  procedures).  With  respect  to  “ease  of  use”  issues,  users 
provided  feedback  on  the  extent  to  which  content  and  format  of  information  displays  is  easily 
interpreted,  as  well  as  on  the  ease  with  which  information  is  located  via  the  main  menu  bar, 
browser  tool,  and  pull-down  lists.  Users  also  provided  their  perceptions  on  the  ease  with  which 
customizable  user  settings  are  adjusted. 

In  considering  the  usefulness  of  LOCIS  as  a  decision  support  tool,  users  assessed  the  extent  to 
which  LOCIS  information  displays  (1)  maintain  situation  awareness,  (2)  provide  information 
relevant  to  job  requirements,  and  (3)  provide  timely  information.  Users  also  assessed  the  extent 
to  which  LOCIS  customization  features  helped  to  “push”  them  the  right  information. 

As  a  part  of  each  Block  Release  evaluation  session,  the  evaluation  team,  led  by  AFRL/HESR, 
conducted  an  out-brief  with  Hurlburt  leadership. 

During  the  final  Block  Release  evaluation  period  conducted  in  mid-June,  users  who  had 
participated  in  at  least  two  of  the  three  evaluation  sessions  were  asked  to  complete  a 
questionnaire.  A  technical  point  of  contact  at  Hurlburt  identified  an  appropriate  set  of  users  and 
requested  inputs  from  them.  Completed  questionnaires  Were  delivered  to  AFRL/HESR. 

The  following  features  were  targeted  for  Block  Release  evaluation: 

a.  User  Interface  Framework 

b.  Information  Displays 

c.  Customization 

d.  LOCIS  Pop-Up  Windows 

e.  Automated  Notifications 

Included  in  the  User  Interface  Framework  were  log  on  procedures  and  navigation.  Information 
Displays  included  Schedule,  Status,  and  Supply  views.  Customization  included  threshold 
settings,  display  options  (for  customization  of  individual  LOCIS  views),  the  workspace  builder, 
and  user  profile  settings.  LOCIS  Pop-Up  Windows  included  the  drill-down  information 
displayed  upon  a  user’s  selection  of  aircraft  icons,  sortie  and  maintenance  blocks,  and  aircraft 
availability  threshold  messages.  Automated  Notifications  included  the  alert/waming  indicator 
(displayed  in  the  menu  bar  and  activated  when  an  aircraft  availability  threshold  is  violated),  the 
alert/waming  log,  and  the  general  message  log. 
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B.2.1.2  Spiral  2  Data  Collection  Approach 

A  single  evaluation  session,  conducted  by  members  of  the  Human  Factors  team,  was  held  in 
conjunction  with  the  Spiral  2  Demonstration  held  at  BAE  SYSTEMS.  Members  of  the 
Operational  Architecture  IPT  provided  functional  support.  During  this  evaluation  session,  eight 
members  of  the  LUG  provided  feedback  on  the  features  demonstrated  during  the  Spiral  2 
Demonstration.  As  in  the  Block  Release  evaluation,  user  data  was  collected  within  the  context  of 
small  group  interview  sessions.  The  eight  LUG  members  were  divided  into  two  groups  of  four 
individuals  each,  and  two  separate  interview  sessions  were  conducted  in  parallel.  One  of  the 
parallel  sessions  focused  on  navigation  (User  Interface  Framework),  schedule  and  supply  views 
(Information  Displays),  and  the  workspace  builder  (Customization),  while  and  the  other  focused 
on  status  views  (Information  Displays),  threshold  settings  (Customization),  LOCIS  notifications 
(Automated  Notifications),  and  the  Tools  link  (User  Interface  Framework).  The  Human  Factors 
team  conducted  two  parallel  sessions  in  the  morning  of  30  April  and  two  parallel  sessions  in  the 
afternoon. 

As  in  the  small  group  sessions  conducted  during  Block  Release  data  collection  activities,  each 
user  group  was  interviewed  on  various  usability  aspects  of  the  set  of  LOCIS  features  pre-selected 
for  the  session.  LUG  feedback  specific  to  each  feature  was  recorded.  LUG  users  also  provided 
subjective  ratings  on  targeted  LOCIS  features. 

B.2.2  LOCIS  Block  Release  Evaluation  Findings 

B.2.2.1  Summary  of  LOCIS  Block  Release  Data  Collection  Sessions:  22-25  JAN  02 

Two  members  of  the  Human  Factors  team  and  one  member  of  the  Operational  Architecture  IPT 
interviewed  8  users  out  of  the  33  trained  in  November  2001.  Of  these  eight  users,  five  had 
logged  back  on  to  LOCIS  at  least  once  after  being  trained. 

The  focus  of  this  evaluation  trip  was  to  collect  feedback  on  log  in/out  procedures  and  on  LOCIS 
navigation  features.  In  general,  users  provided  positive  comments  about  navigation  and  data 
presentation.  Users  considered  the  log  in  procedure  (e.g.,  typing  LOCIS  URL,  selecting  a 
Favorites  link,  selecting  a  LOCIS  desktop  icon)  to  be  straightforward.  One  user  established  his 
browser  home  page  to  be  the  LOCIS  log  on  page. 

The  LOCIS  evaluation  team  encouraged  Hurlburt  participants  to  continue  their  use  of  LOCIS  so 
they  would  be  prepared  to  provide  another  set  of  comments  during  its  second  trip  in  mid- 
February. 
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B.2.2.1.1  General  Comments  from  Users 


Some  users  had  built  additional  workspaces  (i.e.,  in  addition  to  those  built  during  training 
sessions).  These  new  workspaces  were  used  primarily  to  monitor  status  data— primarily  at  the 
beginning  of  the  day— to  provide  a  “heads  up”  on  what  would  need  attention  throughout  the  day. 

Users  indicated  that  “getting  accustomed  to”  the  data  presentation  provided  in  LOCIS  played  a 
big  part  in  their  levels  of  use.  In  other  words,  they  are  accustomed  to  using  the  local  C2  database 
and  are  comfortable  with  it.  LOCIS  is  still  an  unknown  product,  and  users  need  continued 
assurance  that  its  reliability  will  be  consistent  with  the  tools  they  use  now.  They  need  to 
establish  a  “comfort  zone”  (and/or  know  that  their  leadership  is  also  using  LOCIS). 

Users  suggested  a  few  additional  data  items  from  the  C2  database  that  might  be  included  in 
LOCIS  recap  and  supply  views. 

B.2.2.1.2  General  Comments  from  Hurlburt  Leadership 

An  out-brief  with  the  16  SOW  MXG/CC  was  conducted  on  24  January  2002.  LOCIS  received 
strong  support  from  16  SOW  leadership.  The  MXG/CC  believes  the  features  currently  available 
in  the  LOCIS  Block  Release  are  useful.  He  identified  four  issues  that,  once  addressed,  would 
help  motivate  users  to  consider  LOCIS  as  a  useful  work  tool.  (Other  Hurlburt  users  also  raised 
these  issues  during  their  interview  sessions.) 

a.  Clicking  on  the  “x”  in  the  upper  right  comer  of  a  LOCIS  web  page  as  a  means  of  logging 
out  continues  to  be  a  problem.  As  we  have  been  made  aware,  “x”-ing  out  does  not  result 
in  a  log  out;  however,  users  are  unaware  that  when  they  “x”  out,  they  have  not  logged  out. 
Currently,  they  have  no  way  to  recover  from  an  “x”  out.  Note  that  “x”-ing  out  of  a  web 
page  is  a  common  technique  and  among  most  users  is  done  by  habit. 

b.  Eliminating  the  “time  out”  feature  would  be  helpful.  Many  users  would  like  to  keep 
LOCIS  up  and  running  throughout  the  day,  without  being  required  to  repeatedly  log  on 
after  the  time  out  occurs. 

c.  Providing  a  re-size  capability  allowing  users  to  fit  LOCIS  pages  so  that  they  completely 
fill  the  display  area  of  the  monitor  (i.e.,  “maximize”)  would  be  a  definite  advantage.  The 
16  SOW  LG,  for  example,  uses  a  flat  panel  17"  monitor.  Currently,  the  LOCIS  pages  use 
only  approximately  60%  of  the  real  estate  provided  by  his  monitor,  and  the  pages  cannot 
be  expanded. 


d.  Improving  the  notification  users  receive  when  an  agent  is  not  running  (to  ensure  they  are 
made  aware  that  respective  data  are  not  necessarily  valid)  would  also  be  advantageous. 

B.2.2.2  Summary  of  LOCIS  Block  Release  Data  Collection  Sessions:  19-22  FEB  02 

Dining  this  evaluation,  members  of  the  Human  Factors  team  and  Operational  Architecture  IPT 
interviewed  12  users.  (Five  of  these  individuals  provided  feedback  during  the  January 
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evaluation.)  In  addition  to  these  12  users,  the  LOCIS  team  spoke  with  Hurlburt  leadership  (16 
SOW  MXG/CC  and  MXG/CD). 

B.2.2.2.1  General  Comments  from  Users 

By  and  large,  users  like  the  visualization  features  designed  into  LOCIS.  Evaluators  still  received 
comments  about  the  scrolling  requirements  for  schedule  and  status  displays.  If  some  means  of 
allowing  users  to  compress/collapse  schedule  displays  (and  reduce  horizontal  scrolling 
requirements)  could  be  devised,  many  of  the  negative  comments  received  about  scrolling  could 
be  addressed. 

Some  additional  (minor)  “tweaks”  to  the  Recap  display  were  suggested.  Settings  to  adjust  “alert” 
and  “warning”  levels  for  the  A/C  availability  threshold  appear  to  be  confusing  a  number  of  users. 
Specifically,  users  expressed  some  confusion  about  what  the  levels  actually  are  once  they  are  set. 
Some  type  of  print  capability  was  suggested  by  a  number  of  users.  Several  users  strongly 
encouraged  the  LOCIS  team  to  begin  training  Pro  Supers. 

B.2.2.2.2  General  Comments  from  Hurlburt  Leadership 

Again,  in  general,  feedback  was  positive.  Both  the  16  MXG/CC  and  MXG/CD  had  favorable 
comments.  The  16  MXG/CC  indicated  that  if  he  could  make  changes  to  LOCIS,  he  would  make 
two  changes: 

a.  Change  the  default  dimensions  of  each  pop-up  window  to  be  “maximized.” 

b.  Address  the  “x”  out  problem  for  pop-up  windows.  Currently,  when  pop-ups  are  “x”-ed 
out  (rather  than  closed  with  the  "close"  button  provided  in  the  window),  their 
corresponding  icons  accumulate  along  the  lower  tool  bar. 

B.2.2.3  Summary  of  LOCIS  Block  Release  Data  Collection  Sessions:  19-22  MAR  02 

During  this  evaluation,  members  of  the  Human  Factors  team  and  Operational  Architecture  IPT 
interviewed  11  users.  Of  these  11,  three  were  individuals  not  interviewed  during  the  two 
previous  trips.  An  out-brief  with  the  16  SOW  MXG/CC  was  conducted  on  21  March. 

Data  collection  interviews  focused  on  the  following: 

a.  Supply  views 

b.  Unit-level  At-a-Glance  views 

c.  Workspace  builder 

d.  Alerts/wamings  (including  the  A/C  availability  threshold  pop-up  window) 

Again,  comments  were  positive,  although  widespread  use  of  LOCIS  is  still  rather  low. 
Approximately  three  to  four  individuals  (including  16  MXG/CC)  log  on  to  LOCIS  on  a  daily 
basis. 
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B.2.2.3.1  General  Comments  from  Users 

The  recent  updates  BAE  SYSTEMS  incorporated  into  the  All  MICAPs  views  were  well  received. 
Users  liked  the  changes.  (Specifically,  “Urgency”  and  “Status”  fields  were  added,  and  the  two 
“EDD”  fields  were  replaced  with  a  single  “EDD”  field).  Users  provided  a  few  recommendations 
for  additional  data  for  All  MICAPs  and  Supply  Drivers  views. 

Users  like  the  unit-level  At-a-glance  views.  The  geographic  layout  is  useful — particularly  for 
MOC  users.  Showing  aH  A/C  parked  in  specific  locations  (rather  than  displaying  only  those  A/C 
belonging  to  a  respective  unit)  would  be  helpful. 

The  workspace  builder  is  useful,  and  users  like  having  the  ability  to  integrate  different  types  of 
data  into  a  single  “window”.  However,  they  would  also  like  having  control  of  individual  panels 
making  up  a  workspace.  That  is,  making  each  panel  an  independent  entity  would  allow  users 
greater  flexibility  in  controlling  their  workspace  views  (e.g.,  for  resizing  purposes) — especially 
for  four-pane  workspaces  in  which  dimensions  of  individual  panels  are  reduced  significantly. 
Users  also  suggested  allowing  the  addition  of  a  single  view  to  the  list  of  workspaces  (in  addition 
to  the  two-pane  and  four-pane  options  provided  in  the  wizard).  Such  a  feature  would  provide 
users  a  “favorites”  list  and  would  allow  them  to  access  these  favorites  without  using  the 
navigation  bar. 

Allowing  users  to  monitor  threshold  levels  was  regarded  as  useful,  although  given  current 
deployed  conditions,  the  A/C  availability  threshold  is  not  particularly  meaningful.  Evaluators 
received  a  few  suggestions  for  additional  thresholds  (e.g.,  ETIC  “busts”  or  EDD  updates). 

B.2.2.3.2  Comments  from  Hurlburt  Leadership 

Feedback  from  the  16  MXG/CC  was  positive.  The  inclusion  of  additional  (key)  data  in  the  icon 
rollovers  was  suggested.  (These  critical  data  items  are  currently  included  in  A/C  icon  drill 
down.)  Preference  is  to  see  these  critical  data  items  (e.g.,  ETIC,  driving  discrepancy,  TO  and 
land  times)  as  part  of  the  rollover — without  having  to  drill  down  to  access  them. 

The  16  MXG/CC  would  also  find  a  display  showing  flyers  scheduled  across  the  wing  (i.e.,  a 
wing-level  flying  schedule)  useful. 

There  is  some  interest  in  providing  an  LOCIS  intro  briefing  and/or  training  to  the  OG  side  of  the 
house. 

B.2.2.4  Block  Release  Findings:  Details 

This  section  offers  further  details  on  the  feedback  collected  from  users  during  the  three  Block 
Release  evaluation  sessions. 

B.2.2.4.1  User  Interface  Framework 

Users  indicated  that  the  LOCK  interface  was  easy  to  use  and  that  navigation  within  LOCK  was 
straightforward.  No  clear  preference  for  either  navigation  technique  (pull-down  lists  versus 
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organization  chart  accessible  via  the  browser  tool)  was  evident.  That  is,  the  use  of  pull-down 
lists  versus  use  of  the  browser  tool  was  equally  divided  among  users  interviewed.  Some  users 
believed  pull-down  lists  were  awkward  to  use.  Organizations  at  lower  levels  in  the  hierarchy 
cannot  be  directly  selected  from  the  pull-down  lists  unless  corresponding  higher-level 
organizations  have  already  been  selected  from  pull-down  lists.  In  other  words,  to  select  a  view 
for  a  given  organization  at  level  L,  a  user  must  first  select  all  corresponding  higher-level 
organizations  linked  to  L.  For  example,  to  access  the  daily  flying  schedule  for  the  15  AMU  (unit 
level),  the  user  must  first  select  16  MXG  (group  level)  and  16  AGS  (squadron  level)  from  the 
pull-down  lists.  (The  15  AMU  is  one  of  the  three  unit-level  organizations  comprising  the  16 
AGS,  and  the  16  AGS  is  a  squadron-level  organization  within  the  16  MXG.)  Users  suggested 
that  any  organization,  at  any  hierarchical  level,  be  accessible  at  any  time. 

Users  also  commented  on  the  “look  and  feel”  of  the  blue  page  tabs  incorporated  in  LOCIS. 
Although  tab  labels  were  meaningful,  some  users  suggested  enhancements  to  the  “look  and  feel” 
of  LOCIS  tabs  (e.g.,  modifying  tab  appearance  to  resemble  “folder-type”  tabs).  Specifically, 
users  suggested  that  the  hyperlink  capability  associated  with  each  tab  was  not  readily  apparent. 
(Insufficient  contrast  between  the  light  blue  color  of  the  tab  and  the  background  color  of  LOCIS 
pages  contributed  to  this  deficiency.)  In  other  words,  as  displayed  on  a  LOCIS  page,  tabs  were 
not  immediately  recognizable  as  active  links.  Without  training,  users  might  not  recognize 
LOCIS  tabs  as  active  links.  One  user  suggested  modifying  tab  appearance  to  resemble  “folder¬ 
like”  tabs. 

Users  also  agreed  that  a  “single  log  on”  approach  (i.e.,  accessing  LOCIS  via  log  on/password 
entry  to  AF  Portal)  was  desirable  but  acknowledged  that  sensitivity  of  the  data  provided  through 
LOCIS  might  warrant  some  type  of  additional  password  protection. 

Users  employed  a  number  of  different  log  on  techniques — specifically,  inclusion  of  LOCIS  URL 
among  the  list  of  Favorites  (Internet  Explorer),  manual  entry  of  LOCIS  URL  at  each  log  on, 
inclusion  of  LOCIS  URL  to  My  Links  (AF  Portal),  and  specifying  the  LOCIS  log  on  page  as  the 
browser  home  page. 

During  all  evaluation  sessions,  users  expressed  the  need  for  a  print  capability. 

In  addition  to  offering  feedback  during  the  small  group  interview  sessions,  users  also  provided  a 
series  of  subjective  ratings  specific  the  user  interface  framework.  Each  rating  was  based  on  a 
five-point  scale.  Results  from  the  subjective  ratings  portion  of  the  evaluation  are  summarized  in 
the  following  tables.  Each  table  reports  the  number  of  users  assigning  a  given  rating. 
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When  navigating  within 
browser  tool  over  the  dro 

LOCIS,  I  preferred  using  the 
Mown  lists. 
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When  using  the  menu  bar,  desired  categories  of  information 
(e.g.,  Status,  Schedule)  were 
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As  indicated  by  the  subjective  data,  the  ease  of  use  attributed  to  the  user  interface  framework- 
including  navigation,  accessibility  to  the  LOCIS  web  site,  and  locating  desired  information— was 
high.  With  respect  to  navigation  technique  (i.e.,  browser  tool  versus  pull-down  lists),  most  users 
indicated  no  preference  for  one  over  the  other.  On  the  other  hand,  two  users  tended  toward  a 
preference  for  pull-down  lists,  and  one  user  tended  toward  a  preference  for  the  browser  tool. 

B.2.2.4.2  Information  Displays 

In  general  users  placed  a  high  value  on  the  visualization  techniques  (e.g.,  color  coding, 
geographic  layout  of  unit-level  aircraft  status  views,  timeline  format  for  schedule  data)  employed 
in  LOCIS.  Specifically,  users  perceived  the  geographic  layout  applied  in  unit-level  status 
displays  to  be  useful— particularly  for  MOC  users.  Some  users  identified  a  need  to  view 
projected  schedule  .data,  in  addition  to  daily  schedule  data.  Specific  projected  periods  of  time 
(i.e.,  72  hours  or  a  week  in  advance)  were  suggested.  On  the  other  hand,  other  users  preferred 
having  an  option  for  a  user-specified  period  of  time. 

The  All  MICAPs  view  was  revised  in  accordance  with  user  feedback  collected  during  the  second 
evaluation  session.  Specifically,  two  fields  (“Urgency”  and  “Status”)  were  added  to  the  All 
MICAPs  table.  In  addition,  the  two  “EDD”  fields  (intended  to  allow  users  to  track  original  and 
current  (updated)  estimated  delivery  dates)  were  replaced  with  a  single  “EDD”  field.  This  single 
field  is  intended  to  indicate  current  EDD  (and  will  be  updated  as  EDD  is  updated). 

Scrolling  requirements  (particularly  horizontal  scrolling  requirements)  for  the  wing-level  status 
and  schedule  views  were  regarded  as  “annoying.”  (Users  did  indicate  their  preference  for 
vertical  scrolling  versus  horizontal  scrolling.) 

Users  found  the  Data  Sources  view,  accessible  from  the  Tools  link,  to  be  a  useful  means  of 
verifying  the  timeliness  of  LOCK  data. 

Results  from  the  subjective  ratings  portion  of  the  evaluation  are  summarized  in  the  following 
tables.  Each  table  reports  the  number  of  users  assigning  a  given  rating.  In  some  instances, 
ratings  solicited  in  the  February  evaluation  sessions  were  solicited  once  again  in  the  March 
sessions.  In  this  manner,  the  LOCK  team  could  assess  the  extent  to  which  subjective  ratings  on 
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the  usefulness  of  LOCIS  information  displays  changed  between  February  and  March — as  users 
became  more  experienced  with  LOCIS  capabilities.  (Note  that  while  some  of  the  February 
participants  also  returned  for  the  March  evaluations,  not  all  returned  to  provided  feedback  in 
March.)  Repeated  ratings  are  identified  as  such. 

In  general,  users  rated  schedule,  status,  and  supply  views  as  “easy  to  understand”.  When 
considering  the  extent  to  which  LOCIS  information  displays  supported  job  requirements,  all  of 
the  March  participants  agreed  (indicated  via  a  rating  of  1  or  2)  that  information  displays 
supported  tasks  required  by  their  jobs.  The  February  data,  however,  indicates  a  lower  percentage 
of  participants  believed  that  information  displays  supported  job  tasks. 


1  Information  provided  in  LOCIS 

schedule  displays  was  ! 
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5 

Not  at  all  Easy  to  Understand 

4* 
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8 

1  [ 

2 
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0 

Information  provided  in  LOCIS  status  displays  (16  MXG,  16  AGS) 
was  _ 


Not  at  all  Easy  to  Understand 


0 


Information  provided  in  LOCIS  status  displays — i.e.,  geographic 
lavout  displavs  for  4, 15, 16  AMU  (16  AGS)  was  _ 
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Information  provided  in  LOCIS  supply  displays  was  _ 
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LOCIS  information  displays  supported  tasks  required  by  my  job. 


2  3  4  5 

Strongly  Disagree _ 


2*  I  5*  4*  1*  1  0 _ _ 


J _ 16  10  10  10 _ | 

*  Data  collected  during  the  19-22  February  2002  sessions.  All  other  ratings  were  collected  during  the  19-22  March  2002 
sessions. 
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B.2.2.4.3  Customization 


As  one  means  of  creating  customized  workspaces— and  integrating  different  kinds  of  data  into  a 
single  work  area — the  Workspace  Builder  was  viewed  as  a  useful  tool.  Users  agreed  that  having 
the  flexibility  to  place  multiple  views  into  a  single  “space”  allowed  them  to  build  an  information 
display  that  could  provide  a  meaningful  overview  of  current  conditions.  Users  suggested 
including  a  single-view  workspace  into  the  builder,  in  addition  to  the  two-pane  and  four-pane 
workspaces  provided  as  options.  Given  the  option  of  including  single-view  workspaces  to  their 
list  of  customized  workspaces,  users  could  go  directly  to  their  favorite  (single)  views  without 
having  to  navigate  with  the  main  menu  bar. 

Users  expressed  a  desire  for  greater  flexibility  in  controlling  the  contents  (data  elements) 
displayed  in  individual  views.  Their  suggestion  was  to  provide  options  for  turning  on  (or  off) 
any  data  field  displayed  in  a  given  view. 

For  displaying  schedule  data,  users  preferred  view  options  allowing  them  to  display  the  longest 
“window”  of  time  with  respect  to  current  time.  Consequently,  the  typical  display  option  selected 
for  the  schedule  timeline  was  “±12  hours.” 


Results  from  the  subjective  ratings  portion  of  the  evaluation  are  summarized  in  the  following 
tables.  Note  that  each  table  reports  the  number  of  users  assigning  a  given  rating. 


In  general,  users  rated  LOCIS  customization  features  positively,  although  of  the  12  users  who 
considered  customization  as  a  whole,  4  neither  strongly  agreed,  nor  strongly  disagreed  that  such 
features  helped  to  “push”  them  the  right  information.  The  Workspace  Builder — specifically,  its 
ability  to  “push”  users  the  right  information  and  the  manner  in  which  it  leads  users  through  the 
process  of  creating  customized  workspaces — received  positive  ratings.  Of  the  11  users  who 
rated  customization  features,  10  perceived  the  threshold  settings  to  be  easily  adjustable. 
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The  process  of 
straightforward. 
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customized  workspaces  was 

1 

Strongly  Agree 

2 

3 

4 

5 

Strongly  Disagree 

8 

T 

jP 

~ 

0 

B.2.2.4.4  LOCIS  Pop-Up  Windows 

While  users  found  the  detailed  data  contained  in  pop-up  windows  (displayed  as  a  result  of  a 
drill-down  action)  to  be  appropriate,  they  suggested  that  some  of  the  more  critical  elements  be 
included  as  part  of  the  rollover  data  currently  associated  with  aircraft  icons,  sortie  blocks,  and 
maintenance  blocks.  Providing  this  additional  rollover  information  would  reduce  the  need  for 
users  to  perform  many  of  the  drill-down  actions  currently  required  for  retrieval  of  critical  data. 

Users  suggested  that  LOCIS  designers  revisit  the  default  dimensions  specified  for  pop-up 
windows.  The  suggestion  was  to  automatically  adjust  window  dimensions  such  that  match  the 
amount  of  data  displayed  and  eliminate  (or  minimize)  the  need  for  scrolling. 


Results  from  the  subjective  ratings  portion  of  the  evaluation  are  summarized  in  the  following 
table.  The  table  reports  the  number  of  users  assigning  a  given  rating.  Ratings  specific  to  the 
extent  to  which  pop-up  windows  supported  job  tasks  were  solicited  in  both  the  February  and 
March  evaluation  sessions.  (Again,  while  some  of  the  February  participants  also  returned  for  the 
March  evaluations,  not  all  returned  to  provided  feedback  in  March.) 


Information  provided  in  drill-down  pop-up  windows  supported 
tasks  required  by  my  job. 

1 

Strongly  Agree 

2 

3 

4 

5 

6* 

1* 

5* 

0 

0 

5 

6 

0 

0 

0 

All  other  ratings  were  collected  during  the  19-22  March  2002 


sessions. 


When  considering  the  ability  of  information  contained  in  pop-up  windows  to  support  required 
job  tasks,  user  ratings  differed  between  February  and  March.  Specifically,  all  of  the  March 
participants  agreed  (indicated  by  a  rating  of  1  or  2)  that  pop-up  information  supported  tasks 
required  by  their  jobs.  The  February  data,  however,  indicates  users  were  less  inclined  to  agree 
that  pop-up  information  supported  job  tasks. 


B.2.2.4.5  Automated  Notifications 


Users  believed  the  aircraft  availability  threshold  would  be  most  useful  under  non-deployed 
conditions — when  the  majority  of  Hurlburt  aircraft  are  on  site.  They  also  identified  additional 
parameters  that  would  be  useful  to  monitor— specifically,  thresholds  allowing  users  to  be  alerted 
to  ETIC  “busts”  and  EDD  bumps  (for  MOC  users). 

For  the  General  Message  Log,  users  suggested  a  set  of  canned  messages. 
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Results  from  the  subjective  ratings  portion  of  the  evaluation  are  summarized  in  the  following 
table.  This  table  reports  the  number  of  users  assigning  a  given  rating. 


Information  provided  in  the  Alert/Warning  Message  Loe  was.  1 

1 

Useful  in  Informing 
me  of  Current 
Conditions 

2 

3 

4 

5 

Not  at  all  Useful  in 
Informing  me  of  Current 
Conditions 

3 

3 

4 

1 

When  assessing  the  usefulness  of  message  log  information  in  maintaining  situation  awareness 
users  expressed  some  level  of  doubt.  Of  the  11  who  rated  the  quality  of  Alert/Waming  Message 

Log  information,  4  users  were  unsure  of  its  usefulness,  and  1  user  found  it  to  be  “not  at  all 
useful.” 

i 

B.2.2.5  Final  Block  Release  Evaluation 

The  questionnaire  developed  for  the  final  evaluation  period  consisted  of  nine  questions  designed 
to  capture  users’  subjective  ratings  of  LOCIS  navigation  features,  information  displays  and  pop¬ 
up  windows,  customization  features,  and  notifications.  The  intent  of  this  particular  evaluation 
was  to  focus  users’  assessments  on  LOCIS  as  a  “complete  package”  (i.e.,  to  assess  a  set  of 
capabilities,  rather  than  the  content  and  format  of  individual  information  views  or  pop-up 
windows).  Consequently,  the  ratings  designed  for  the  final  evaluation  were  more  general  in 
nature  than  those  developed  for  each  of  the  monthly  evaluation  sessions.  Each  rating  was  based 
on  a  five-point  scale,  and  results  are  summarized  in  tables  to  follow.  Each  table  reports  the 
number  of  users  assigning  a  given  rating. 


In  addition  to  addressing  LOCIS  capabilities,  the  questionnaire  also  addressed  LOCIS’  potential 
as  an  operational  support  tool.  Questions  developed  for  this  portion  of  the  questionnaire  were 
designed  to  capture  preliminary  data  on  the  LOCIS  exit  criteria  established  by  AFRL/HESR. 
These  exit  criteria  have  been  defined  as  follows. 


a.  Reduce  the  time  to  re-plan  during  a  crisis-action  contingency  by  20%. 

b.  Reduce  staff  hours  to  produce  capability  and  historical  reports  by  25%. 

c.  Reduce  the  time  required  for  an  assessment  of  wing/unit  level  capability  by  25%. 

Eleven  users  completed  the  questionnaire  during  the  final  evaluation  period.  To  provide 
evaluators  with  information  on  users’  experience  levels,  the  number  of  times  a  user  had  logged 
on  to  LOCIS  since  January  02  was  recorded  on  his  respective  questionnaire.  This  number  ranged 
from  3  to  more  than  250,  and  two  distinct  user  groupings  were  immediately  evident.  A  small 
group  of  users  had  logged  more  than  60  LOCIS  sessions  since  January,  and  a  larger  group  of 
users  had  logged  a  maximum  of  only  10  sessions.  Consequently,  each  user’s  experience  level 
was  defined  in  terms  of  one  of  these  two  groups.  The  more  experienced  users  were  those  who 
logged  more  than  60  LOCIS  sessions  (Group  A),  while  the  less  experienced  users  (Group  B)  had 

logged  no  more  than  10  sessions.  Four  users  were  identified  as  Group  A  users,  and  seven  were 
assigned  to  Group  B. 
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Because  of  the  large  discrepancy  between  the  number  of  sessions  logged  by  Group  A  users  and 
those  logged  by  Group  B  users,  the  Human  Factors  team  believed  experience  level  could  have 
some  influence  on  user  responses,  and  any  experience-based  differences  should  be  identified. 
Consequently,  user  data  are  reported  according  to  experience  level,  where  ratings  collected  from 
Group  A  users  are  reported  separately  from  Group  B  ratings.  In  this  manner,  any  differences 
between  the  two  groups  can  be  readily  identified. 

B.2.2.5.1  Navigation 

Users’  perceptions  regarding  the  straightforwardness  of  LOCIS  navigation  tasks  are  summarized 
in  the  following  table. 


Navigation  within  LOCIS  is  a  straightforward  task. 

1  2  I  3  4  7  5 

Strongly  A  gree  Strongly  Disagree 


3  10  0  0 


2  I  4  |  1  |  0  I  0 


As  indicated  through  the  summary  data,  regardless  of  experience  level,  most  users  found 
navigation  within  LOCIS  to  be  straightforward. 

B.2.2.5.2  LOCIS  Information  Displays  and  Pop-Up  Windows 

Through  a  series  of  four  ratings,  users  provided  feedback  on  the  extent  to  which  LOCIS  displays 
and  detailed  drill-down  information  maintained  situation  awareness.  Specifically,  users  were 
asked  to  rate  the  following: 

a.  How  well  LOCIS’  real  time  data  facilitated  an  understanding  of  current  conditions 

b.  The  extent  to  which  LOCIS  visualization  techniques  helped  to  identify  critical  issues 

c.  The  ease  with  which  information  displays  were  interpreted 

d.  The  value  of  the  detailed  information  provided  as  a  drill-down  capability. 

User  inputs  are  summarized  in  the  following  four  tables. 


The  real-time  nature  of  LOCIS  data  allows  users  to  maintain  an 
understanding  of  current  conditions. 

1 

Strongly  Agree 

2 

3 

4 

5 

2 

BOD 

0 

2 

LL 

D 

0 

LOCIS  visualization  features  (e.g.,  timeline  formats  for  schedule 
data,  color-coded  aircraft  icons,  geographic  layouts)  help  to 
highlight  issues  requiring  attention. 


1  2  3  4  5 

Strongly  Agree _  _  Strongly  Disagree _ 


3  10  0  0 


3  3  10  0 
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The  ratings  data  indicate  that  most  users,  regardless  of  experience  level,  believed  LOCIS  did  a 
good  job  of  supporting  situation  awareness,  although  one  less  experienced  user  was  dissatisfied 
with  the  usefulness  of  drill-down  data.  When  considering  the  ability  of  LOCIS’  real  time  data 
and  visualization  techniques  to  help  identify  critical  situations,  one  Group  B  user  indicated, 
through  a  rating  of  3,  no  specific  benefit  to  either  of  these  features.  Similarly,  when  considering 
the  ease  with  which  status,  schedule,  and  supply  data  are  interpreted,  one  user  indicated  no 
specific  advantages  of  the  presentation  features  provided  in  LOCIS  displays  and  pop-up 
windows. 

B.2.2.5.3  Customization  Features 

Through  a  series  of  three  ratings,  users  assessed  LOCIS’  primary  customization  features — 
specifically,  the  Workspace  Builder  and  the  Options  button  available  in  LOCIS  information 
views.  (The  Options  button  allows  users  to  change  the  look  and  feel  of  individual  information 
views.) 


The  Workspace  Builder  is  useful  because  it  allows  users  to  integrate 
information  most  critical  to  them. 

1  I  2  I  3  I  4  I  5 

Agree  Strongly  Disagree 


Group  B  users  rated  LOCIS  customization  features  less  favorably  than  they  rated  navigation  and 
information  display/pop-up  window  features.  With  respect  to  the  three  customization  aspects 
evaluated,  the  Workspace  Builder’s  usefulness  in  allowing  users  to  integrate  critical  information 
was  rated  least  favorably.  Three  of  the  seven  Group  B  users  found  the  task  of  creating 
workspaces  to  be  less  than  straightforward,  and  three  Group  B  users  were  not  convinced  the 
flexibility  of  the  Options  feature  was  sufficient  to  support  information  display  customization.  In 
comparing  customization  feature  ratings  between  Group  A  and  Group  B,  a  larger  proportion  of 
Group  B  users  were  dissatisfied  with  LOCIS  customization  features.  The  less  favorable  inputs 
from  Group  B  users  might  be  attributed  to  their  lower  experience  levels.  With  more  extensive 
LOCIS  interaction,  these  ratings  might  be  improved. 

B.2.2.5.4  LOCIS  Notifications 

Users  were  asked  to  evaluate  the  usefulness  of  LOCIS  notifications  in  maintaining  situation 
awareness.  Of  specific  interest  in  this  evaluation  was  the  Alert/Waming  Message  Log. 


Information  available  in  the  Alert  /  Warning  Message  Log  is  useful 

in  keeping  users  aware  of  current  conditions. 

2 

3 

4 

5 

Strongly  Disagree 

Group  A 

2 

El 

a 

o 

0  ^ 

Group  B 

2 

J] 

a 

a 

0 

The  ratings  data  indicate  that  slightly  more  than  33%  of  users  expressed  some  doubt  that  the 
Alert/Waming  Message  Log  was  useful  in  maintaining  situation  awareness.  In  comparing  ratings 
between  Group  A  and  Group  B,  a  larger  proportion  of  Group  A  users  expressed  concern  over  the 
usefulness  of  the  Alert/Waming  Message  Log. 

B.2.2.5.5  LOCIS  as  a  Support  Tool 

The  intent  of  the  final  portion  of  the  questionnaire  (“LOCIS  as  a  Support  Tool”)  was  to  collect 
preliminary  data  on  the  three  LOCIS  exit  criteria  established  by  AFRL/HESR.  To  capture  this 
type  of  feedback,  the  Human  Factors  team  developed  the  three  two-part  questions  stated  below. 

a.  Do  you  believe  LOCIS  could  reduce  the  time  required  to  re-plan  during  crisis-action 
contingencies? 

If  yes,  by  what  percentage  do  you  estimate  re-planning  time  could  be  reduced? 

b.  Do  you  believe  LOCIS  could  reduce  the  time  required  to  generate  capability  and 
historical  reports? 

If  yes,  by  what  percentage  do  you  estimate  report  generation  time  could  be  reduced? 
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c.  Do  you  believe  LOCIS  could  reduce  the  time  required  for  assessing  wing-  and  unit-level 
capability? 

If  yes,  by  what  percentage  do  you  estimate  assessment  time  could  be  reduced? 

A  summary  of  responses  to  these  questions  is  recorded  in  the  following  tables.  Users’  estimates 
of  potential  improvements  due  to  LOCIS  are  provided  in  parentheses  in  the  Yes  columns.  Data 
are  reported  according  to  experience  level.  Note  that  not  all  users  responded  to  each  question, 
nor  did  all  offer  estimates  when  responding  with  a  yes. 


Do  you  believe  LOCIS  could  reduce  the  time  required  to  re-plan 
during  crisis-action  contingencies?  (If  yes,  by  what  percent?) 

Yes 

No 

Group  A 

3  (20-30%,  25%,  30%) 

1 

Group  B 

4(25%,  20%,  20%) 

2 

Do  you  believe  LOCIS  could  reduce  the  time  required  to  generate 
capability  and  historical  reports?  (If  yes,  by  what  percent?) 

Yes 

No 

Group  A 

1  (50%) 

3 

Group B 

4  (20%,  25%) 

2 

Do  you  believe  LOCIS  could  reduce  the  time  required  for 
assessing  wing-  and  unit-level  capability?  (If  yes,  by  what 
percent?) 

Yes 

No 

Group  A 

2  (20%,  15%) 

2 

Group  B 

6  (10%,  25%.  25%,  40%) 

1 

With  the  exception  of  LOCIS’  ability  to  support  the  preparation  of  capability  and  historical 
reports,  most  users  believed  LOCIS  could  provide  useful  support — specifically,  in  reducing  the 
time  required  for  re-planning  activities  and  assessments  of  wing-  and  unit-level  capability.  When 
considering  LOCIS’  potential  to  support  capability/historical  report  generation,  the  ratings  were 
split.  Five  users  believed  LOCIS  could  save  time  in  generating  reports,  while  five  believed  that 
no  time  savings  could  be  expected. 

The  three  exit  criteria  establish  the  levels  by  which  AFRL/HESR  expects  LOCIS  to  reduce  the 
time  required  for  re-planning  efforts,  generating  capability  and  historical  reports,  and  assessing 
wing-  and  unit-level  capability.  Specifically,  reductions  of  20%  (re-planning),  25%  (reports), 
and  25%  (capability  assessments)  are  expected.  All  estimates  of  expected  reductions  in 
conducting  re-planning  efforts  were  at  least  20%.  When  considering  LOCIS  support  of  report 
generation  tasks,  one  Group  A  user  estimated  a  50%  reduction  in  task  performance  time; 
however,  the  user  qualified  this  estimate  by  stating  it  could  only  be  realized  with  the  addition  of  a 
LOCIS  print  capability.  Two  of  the  four  Group  A  users  did  not  believe  LOCIS  could  save  time 
in  assessing  wing-  and  unit-level  capabilities.  (The  reason  for  one  of  these  no  responses  was  due 
to  LOCIS’  reliance  on  Hurlburt’s  local  C2  database.) 

In  reviewing  the  exit  criteria  feedback— in  particular,  user  estimates  of  time  savings  that  could  be 
realized  through  LOCIS — note  that  it  is  only  preliminary  and  is  not  based  on  actual  user 
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performance  data.  That  is,  the  estimates  do  not  reflect  comparisons  of  current  task  performance 
times  (for  re-planning,  report  generation,  and  capability  assessment  tasks)  against 
LOCIS-supported  task  performance  times.  Note,  too,  that  most  of  the  estimates  were  provided 
by  less  experienced  users. 

B.2.3  Spiral  2  Demonstration  Evaluation  Findings 

Many  of  the  comments  gathered  during  the  Block  Release  evaluation  sessions  were  reiterated 
during  die  Spiral  2  evaluation  sessions.  Some  of  these  comments  are  repeated  in  the  following 
section  as  a  means  of  validating  Block  Release  data;  however,  the  intent  of  this  section  is  to 
document  user  feedback  not  provided  during  the  course  of  the  Block  Release  evaluation. 

B.2.3.1  Spiral  2  Data  Collection  Sessions 

No  significant  changes  were  recommended  for  any  of  the  views  evaluated,  although  several 
suggestions  were  made  in  terms  of  “tweaking”  or  refining  individual  views  by  adding 
information  or  reformatting  existing  information.  The  addition  of  information  generally  involved 
adding  data  to  rollovers,  or  adding  a  column  to  an  existing  table.  Other  changes  could  be 
accomplished  by  hyperlinking  existing  text  to  existing  pages.  Other  suggestions  included 
making  existing  text  more  prominent  by  incorporating  boldfaced  or  enlarged  text. 

B.2.3.1.1  User  Interface  Framework 

In  considering  the  navigation  options  provided  in  LOCIS,  LUG  members  expressed  no 
preference  for  either  the  pull-down  lists  or  the  browser  tool;  however,  some  members  preferred 
that  only  one  navigation  technique  be  provided  in  LOCIS  (i.e.,  provide  a  pull-down  list  option  or 
a  browser  tool  option  but  not  both). 

To  enhance  the  “interpretability”  of  the  labels  assigned  to  main  menu  links,  LUG  members 
suggested  providing  a  brief  definition  or  description  of  the  information  a  user  might  find 
underneath  each  link.  This  definition  could  be  included  as  part  of  a  rollover  for  the  link.  Along 
these  same  lines,  members  also  suggested  providing,  for  each  category  link,  a  list  of  the 
respective  tab  links — e.g.,  a  rollover  for  the  Schedule  link  (unit  level)  would  list  “All  Aircraft”, 
“Flyers”,  “Maintenance”,  and  “Launch  Sequence.” 

LUG  members  indicated  that  Fleet  Status  (accessible  via  the  Tools  link)  was  not  a  high  priority 
requirement,  although  it  may  be  more  relevant  to  ACC  than  AFSOC.  Users  also  indicated  that 
the  information  provided  might  already  be  present  in  MERLIN  and/or  ALIA. 

As  in  the  Block  Release  evaluation  sessions,  participants  provided  a  series  of  subjective  ratings 
specific  the  user  interface  framework.  Results  are  provided  in  the  following  tables.  The  same 
five-point  rating  scale  was  applied;  however,  in  some  cases,  questions  were  modified  slightly  to 
increase  their  relevance  for  LUG  members.  Results  from  the  subjective  ratings  portion  of  the 
evaluation  are  summarized  in  the  following  tables.  Each  table  reports  the  number  of  users 
assigning  a  given  rating. 
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To  navigate  within  LOCIS,  1  prefer  the  browser  tool  over 
the  drop-down  lists. 

1 

Strongly  Agree 

2 

3 

4 

5 

Strongly  Disagree 

4 

1 

0 

0 

In  the  menu  bar,  desired  categories  of  information  (e.g.. 
Status,  Schedule)  are 

1 

Easy  to  Locate 

2 

3 

4 

5 

Not  at  all  Easy  to  Locate 

3 

4 

0 

i 

0 

Obtaining  desired  fleet  status  information  (accessible  via 
the  Tools  link)  is  straightforward.  * 

1 

Strongly  Agree 

2 

3 

4 

5 

Strongly  Disagree 

1 

$  fina  tiPAp  /ti/t  npf 

_3_ 

2 

i 

0 

*  One  user  did  not  provide  a  rating. 


As  indicated  by  the  subjective  data,  the  ease  of  use  attributed  to  the  main  menu— specifically, 
locating  desired  categories  of  information— was  high,  although  one  LUG  member  found  this 
information  somewhat  more  difficult  to  locate  than  did  the  remaining  members.  When 
considering  the  accessibility  of  data  available  through  the  Tools  link,  however,  participants  were 
less  inclined  to  offer  a  high  ease  of  use  rating.  Specifically,  four  of  seven  participants  considered 
the  task  of  obtaining  desired  fleet  status  information  to  be  straightforward — with  only  one  of 
these  four  strongly  agreeing  that  the  task  was  straightforward.  With  respect  to  navigation 
technique  (i.e.,  browser  tool  versus  pull-down  lists),  five  of  eight  participants  tended  toward  a 
preference  for  the  browser  tool.  The  remaining  three  participants  expressed  no  clear  preference. 

B.2.3.1.2  Information  Displays 

For  status  views,  LUG  members  suggested  making  the  MC  summary  statistics  more  prominent 
by  emphasizing  them  through  bold-faced  text  or  moving  them  to  the  top  of  each  Unit  or 
Squadron  window.  Also,  the  squadron  label  and  MC  rate  text  were  suggested  as  potential 
hyperlinks.  For  example,  at  the  wing  level,  clicking  on  the  label  identifying  a  squadron  or  unit 
would  take  the  user  to  the  squadron  or  unit-level  screens.  By  clicking  on  MC  rate  text,  the  user 
would  be  taken  to  more  detailed  data  for  MC  rate.  Identification  of  “hangar  queen”  status  was 
suggested.  LUG  members  indicated  the  importance  of  knowing  which  aircraft  have  been  out  of 
service  for  a  lengthy  period  of  time.  Members  pointed  out  that  in  addition  to  knowing  hangar 
queen  status,  an  MXG  may  simply  want  to  know  which  aircraft  are  “problem  children”  from  the 
standpoint  of  a  lengthy  NMC  period.  LUG  members  suggested  including  as  much  information  as 
possible  about  transient  and  off  site  aircraft.  If  information  is  not  available,  they  suggested 
“graying  out”  the  aircraft.  The  overall  philosophy  was  to  provide  information  about  each  and 
every  aircraft  on  the  facility,  regardless  of  its  origin  or  owning  agency. 

LUG  members  indicated  that  if  possible,  scrolling  should  be  eliminated.  If  scrolling  cannot  be 
completely  eliminated,  vertical  scrolling  is  more  acceptable  than  horizontal  scrolling. 
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In  assessing  the  view  depicting  pilot  certification  data,  LUG  members  could  not  confirm  its 
usefulness  for  OG-level  decision  making.  A  suggestion  was  made  to  add  maintainer  certification 
status  in  this  area. 


In  considering  LOCIS  flying  schedule  views,  LUG  members  suggested  that  higher  headquarter 
missions  be  identified  as  such  (i.e.,  distinguishing  them  from  “regular”  missions).  The  launch 
sequence  view  (not  provided  as  a  Block  Release  capability)  was  well  received,  although  some 
members  questioned  the  need  for  two  separate  views — suggesting  the  integration  of  launch 
sequence  and  flying  schedule  data  into  a  single  view.  One  suggestion  for  merging  launch 
sequence  and  flying  schedule  data  was  to  add  the  appropriate  flying  schedule  data  to  the  current 
launch  sequence  view.  Another  suggestion  was  to  link  the  launch  sequence  graphic  for  a  specific 
line  number  to  its  respective  sortie  block.  In  other  words,  launch  sequence  information  would  be 
provided  as  part  of  the  drill-down  information  linked  to  sortie  blocks.  Members  commented  that 
to  be  completely  proactive  in  monitoring  flying  schedules,  users  must  not  only  be  alerted  to  those 
instances  in  which  an  ETIC  exceeds  a  take  off  time,  they  must  also  be  notified  of  any  delays  in 
the  launch  sequence.  In  other  words,  a  more  proactive  approach  might  be  to  trigger  notifications 
(alerts)  based  on  launch  sequence  delays. 

When  asked  to  compare  the  merits  of  the  original  All  MICAPs  view  to  the  recently  revised  All 
MICAPs  view  (current  Block  Release  version),  some  members  acknowledged  that  the  “Original 
EDD”  field  provided  in  the  original  All  MICAPs  table  (and  not  included  in  the  Block  Release 
table)  would  be  particularly  meaningful  for  supply  users. 

Results  from  the  subjective  ratings  portion  of  the  evaluation  are  summarized  in  the  following 
tables.  In  general,  most  LUG  members  rated  the  information  depicted  in  schedule  and  supply 
views  as  “easy  to  understand.” 


Information  provided  in  LOCIS  schedule  displays  is  I 

3 

4 

5 

Not  at  all  Easy  to  Understand 

5 

1 

1 

m 

0 

Information  provided  in  LOCIS  supply  displays  is 

1 

Easy  to  Understand 

2 

3 

4 

5 

Not  at  all  Easy  to  Understand 

6 

n 

D 

lil 

0  1 

Information  provided  in  LOCIS  status  displays  (16  MXG,  16  AGS) 
is 

2 

3 

4 

5 

Not  at  all  Easy  to  Understand 

4 

El 

D 

o 

0 

Information  provided  in  LOCIS  status  displays— i.e.,  geographic 
layout  displays  for  the  4, 15,  and  16  AMU — is _ _ 


B.2.3.1.3  Customization 


LUG  members  found  the  Workspace  Builder  to  be  a  useful  tool — suggesting  the  most  useful 
multi-view  workspaces  would  be  those  in  which  the  individual  views  making  up  the  workspace 
represent  aggregations  of  data  (i.e.,  higher-level  data  summaries).  Because  dimensions  of  the 
views  combined  into  a  single  workspace  are  necessarily  reduced,  members  suggested  that  the 
most  meaningful  representation  of  the  data  associated  with  each  individual  view  would  be  an 
“abstract”  (summary)  of  that  data.  While  LUG  members  agreed  with  Block  Release  users  that  a 
single-view  workspace  would  be  a  useful  option  to  include  in  the  builder,  they  also  suggested 
including  a  triple-view  workspace  option.  One  member  provided  an  interesting  twist  to  the 
concept  of  multi-view  workspaces  by  suggesting  a  “picture  in  a  picture”  feature.  Another 
member  suggested  that  the  workspace  builder  concept  be  expanded  to  a  “view  builder”  concept 
in  which  views  are  built  up  from  individual  data  elements  via  queries. 

Members  expressed  some  confusion  when  asked  whether  the  pointer  on  the  threshold  slider  bar 
represented  a  red  or  yellow  level.  LUG  members  suggested  color  coding  the  slider  pointers  to 
indicate  whether  a  selected  number  would  be  assigned  a  red  or  yellow  level. 

As  reflected  in  the  subjective  data  provided  in  the  following  tables,  participants  rated  the 
Workspace  Builder  positively.  Specifically,  its  ability  to  “push”  users  the  right  information  and 
the  manner  in  which  it  leads  users  through  the  process  of  creating  customized  workspaces — 
received  positive  ratings. 


The  Workspace  Builder  helped  to  “push”  the  right  information  to 
users. 

1 

2 

3 

4 

5 

Strongly  Agree 

Strongly  Disagree 

6 

_2_ 

0 

0 

0 

The  process  of  creating  customized  workspaces  is  straightforward.  1 

1 

Strongly  Agree 

2 

3 

4 

5 

Strongly  Disagree 

6 

_2_ 

0 

T 

0 

Threshold  settings  are  | 

1 

Easy  to  Adjust 

2 

3 

4 

5 

Not  at  all  Easy  to  Adjust 

1 

0 

6 

0 

B.2.3.1.4  LOCIS  Pop-Up  Windows 

In  the  discussion  of  thresholds  (aircraft  availability),  LUG  member  identified  MC  rate  as  an 
important  issue,  and  they  indicated  a  desire  to  see  more  data  indicating  what  is  contributing  to 
MC  rate.  For  example,  members  suggested  breaking  down  NMC  rate  into  the  components 
“supply”  “maintenance,”  and  “both.”  Members  also  suggested  the  addition  of  a  threshold  that 
would  allow  users  to  monitor  “total  possessed  hours.” 
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LUG  members  also  expressed  some  concern  that  the  drill-down  information  displayed  when  a 
Recap  view  aircraft  icon  is  selected  represents  current  (real-time)  data,  while  the  nature  of  recap 
data  is  inherently  historical.  Members  suggested  that  displaying  real-time  data  (the  aircraft  drill¬ 
down  information)  within  the  context  of  historical  data  ( Recap  view)  is  a  potential  source  of 
confusion  among  users. 


In  general,  most  members  believed  the  drill-down  information  accessible  from  schedule  and 
supply  views  added  value  to  schedule  and  supply  data  contained  within  those  views. 
Specifically,  seven  of  eight  LUG  participants  agreed  (indicated  by  a  rating  of  1  or  2)  that  drill¬ 
down  information  added  value  to  higher-level  views. 


Drill-down  Information  accessible  from  LOCIS  schedule  displays 
adds  value  to  schedule  data. 

1 

Strongly  Agree 

2 

3 

■ 

3 

m 

1 

0 

0 

Drill-down  information  accessible  from  LOCIS  supply  displays 
adds  value  to  supply  data. 

2 

EB 

I 

4 

wm 

0 

1 

o  1 

Drill-down  information  accessible  from  LOCIS  status  displays  adds 
value  to  status  data. 

1 

Strongly  Agree 

2 

3 

m 

6 

1 

mm 

0 

0  . _J 

Drill-down  information  accessible  from  the  Alert/Warning  Message 
Log  adds  value  to  alerts  and  warnings. 

1 

Strongly  Agree 

2 

3 

El 

4 

3 

1 

0 

0  J 

B.2.3.1.5  Automated  Notifications 

In  considering  the  message  logs  provided  in  LOCIS,  users  indicated  that  grouping  notifications 
by  “subject”  seems  appropriate,  with  an  “action  taken”  category  to  indicate  the  status  of  the 
notification.  The  “green-yellow-red”  color  coding  approach  applied  in  the  Alert/Waming 
Message  Log  caused  some  concern  among  LUG  members,  who  felt  that  it  could  potentially 
confuse  aircraft  availability  levels  (green  -  OK,  yellow  -  Warning,  red  -  Alert)  with  aircraft 
status  (green  -  FMC,  yellow  -  PMC,  red  -  NMC).  In  fact,  several  members  erroneously 
confused  the  setting  of  an  aircraft  availability  threshold  level  with  aircraft  status. 

As  indicated  by  the  subjective  ratings  summarized  in  the  following  table,  most  users  found  the 
information  available  in  the  message  log  easy  to  understand. 
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Information  provided  in  the  Alert/Warning  Message  Log  is  1 

1 

Easy  to  Understand 

2 

3 

4 

5 

Not  at  all  Easy  to  Understand 

4 

1 

_0_ 

0 

B.2.4  Human  Factors  Recommendations 


Based  on  user  feedback  collected  during  Block  Release  and  Spiral  2  Demonstration  evaluations, 
the  following  human  factors  recommendations  are  provided.  The  Human  Factors  team  suggests 
these  recommendations  be  considered  for  future  implementations  of  the  LOCIS  capability, 
including  the  Spiral  3  effort  and  any  future  block  release  versions.  Recommendations  are 
grouped  according  to  the  five  features  targeted  for  evaluation:  User  Interface  Framework, 
Information  Displays,  Customization,  LOCIS  Pop-Ups,  and  Automated  Notifications. 

User  Interface  Framework 

a.  Address  printing  requirements. 

b.  Enhance  the  “look  and  feel”  of  LOCIS  page  tabs  (i.e.,  to  make  them  more  recognizable  as 
active  links). 


Information  Displays 

a.  Address  scrolling  issues  (particularly  for  wing-level  status  view  and  multi-view 
workspaces). 

b.  Fill  any  empty  data  fields  displayed  in  Supply  views. 

c.  Provide  wing-level  schedule  data. 

d.  Show  locations  of  all  aircraft  (in  unit-level  status  views). 

e.  Identify  design  options  for  higher-level  summary  views  of  flying  schedule,  status,  and 
supply  data  to  accommodate  constraints  imposed  by  multi-view  workspaces. 


Customization 

a.  Provide  additional  information  in  rollovers. 

b.  Provide  a  single-view  workspace  as  an  option  in  the  Workspace  Builder— to  provide 
more  direct  access  to  user  “favorites” — and  also  provide  a  triple- view  workspace  option. 

c.  Allow  users  direct  control  of  individual  views  contained  within  a  workspace — 
specifically,  allow  “selection”  and  “resizing”  of  individual  views. 

d.  Allow  users  to  save  format  changes  (e.g.,  supply  views). 

e.  Re-examine  the  current  approach  for  setting  aircraft  availability  threshold  levels. 
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LOCIS  Pop-Ups 

Adjust  the  default  dimensions  of  pop-up  windows — such  that  dimensions  of  a  given  window 
match  the  amount  of  data  displayed  and  eliminate  (or  minimize)  scrolling  requirements. 

Automated  Notifications 

Consider  including  additional  thresholds  (e.g.,  visualization  of  ETIC  “busts”  or  EDD  bumps). 

B.3  LOCIS  Spiral  3  Usability  Evaluation  Results 

The  LOCIS  Spiral  3  Demonstration  was  held  at  Wright-Patterson  AFB  (AFRL/HESR)  on  10 
December  2002.  In  conjunction  with  the  Spiral  3  Demonstration,  AFRL/HESR  hosted  a 
usability  evaluation  on  10-11  December  2002.  During  evaluation  sessions,  LUG  members 
provided  feedback  on  selected  capabilities  of  the  LOCIS  Spiral  3  implementation.  Selected 
capabilities  were  those  either  newly  implemented  for  Spiral  3  or  modified  from  the  Spiral  2 
Block  Release  and  Spiral  2  Demonstration  versions  of  LOCIS. 

The  following  table  summarizes  the  capabilities  targeted  for  the  Spiral  3  usability  evaluation. 


Interface  Feature 

Capability  to  be  Evaluated 

User  Interface  Framework 

•  Navigation 

•  Report  Generator 

LOCIS  Information  Displays 

•  Schedule  Displays 

•  Status  Displays 

•  Munitions  Displays 

•  “Print  Screen” 

Tools 

•  Forecasting  Tool 

LOCIS  Notifications 

•  Threshold  Settings 

•  Alert  and  Warning  Message  Log 

Customization 

•  Workspace  Builder 

The  human  factors  data  collection  effort  was  led  by  GTRI  and  supported  by  UDRI.  BAE 
SYSTEMS  and  AFRL/HESR  provided  functional  expertise.  A  total  of  nine  LUG  members 
provided  feedback.  After  completing  a  training  session,  LUG  members  were  divided  into  two 
groups,  and  feedback  was  provided  within  the  context  of  small  group  interview  sessions.  The 
data  collection  team  was  also  divided  into  two  groups  (one  led  by  GTRI  and  another  by  UDRI) 
so  that  two  parallel  evaluation  sessions  could  be  conducted.  The  GTRI  data  collection  team 
captured  feedback  on  the  following  features: 

a.  LOCIS  Information  Displays  (Munitions  Displays) 

b.  Tools 

c.  LOCIS  Notifications 

The  UDRI  data  collection  team  captured  LUG  feedback  on  the  following  features: 

a.  User  Interface  Framework 

b.  LOCIS  Information  Displays  (Status  and  Schedule  Displays,  “Print  Screen”) 

c.  Customization 
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The  following  paragraphs  summarize  the  major  findings  of  the  Spiral  3  usability  evaluation.  In 
addition  to  offering  feedback  during  the  small  group  interview  sessions,  LUG  members  also 
provided  a  series  of  subjective  ratings  specific  to  the  capabilities  being  targeted  for  the  Spiral  3 
evaluation.  Each  rating  was  based  on  a  five-point  scale.  Results  from  the  subjective  ratings 
portion  of  the  evaluation  are  included  in  the  following  sections. 

B.3.1  Summary  of  LUG  Comments 

In  general,  LUG  feedback  on  the  capabilities  implemented  for  LOCIS  Spiral  3  was  favorable. 
LUG  members  were  impressed  with  the  progress  LOCIS  has  made  since  its  earlier  spirals. 

B.3.1 .1  User  Interface  Framework 

B.3.1 .1.1  Navigation 

In  general,  LUG  members  found  the  navigation  approach  to  be  good.  They  also  believed  the 
organization  chart  icon  displayed  in  the  main  menu  worked  well.  LUG  members  made  several 
recommendations  for  improving  the  “look  and  feel”  of  LOCIS  links.  Specifically,  they  suggested 
greater  contrast  between  the  links’  text  labels  and  the  background  color  of  the  main  menu.  LUG 
members  also  suggested  that  the  Reports  link  be  located  closer  to  the  information  display 
category  links.  They  also  recommended  that  the  LOCIS  tabs  be  modified  to  have  a  more  “tab¬ 
like”  look  and  feel.  The  LOCIS  interface  should  provide  a  clear  indication  that  a  given  tab  has 
been  selected  (e.g.,  highlighted  tab) — meaning  that  its  respective  view  is  currently  displayed. 
(Currently,  this  indication  is  not  always  provided.) 

The  following  table  documents  subjective  ratings  for  the  navigation  approach  implemented 
within  LOCIS.  Each  table  reports  the  number  of  users  assigning  a  given  rating. 


Navigating  within  LOCIS  via  the  browser  tool  is  straightforward.  1 

2 

3 

4 

5 

Strongly  Disagree 

Group  1 

4 

1 

0 

0 

0 

Group  2 

1 

_3_ 

0 

0 

0 

B.3.1. 1.2  Report  Generator 

It  was  difficult  for  participants  to  comment  on  the  report  generation  feature  because  the 
demonstrated  implementation  depicted  only  limited  capabilities.  The  report  generator  must 
allow  for  more  reports  than  just  the  Daily  Flyers  and  Recap. 

The  following  tables  document  subjective  ratings  for  the  LOCIS  report  generation  capability. 
Each  table  reports  the  number  of  users  assigning  a  given  rating. 


Navigating  within  LOCIS  via  the  browser  tool  is  straightforward.  1 

1 

Strongly  Agree 

2 

3 

4 

5 

Strongly  Disagree 

Group  1 

4 

D 

o 

o 

0 

Group  2 

1 

B 

□ 

B 

0 
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The  report  generator  capability  adequately  satisfies  the  need  to 
print  LOCIS  data. 

1 

2 

3 

4 

5 

Strongly  Agree 

Strongly  Disagree 

Group  1 

1 

H 

a 

a 

0 

Group  2 

0 

n 

D 

a 

1 

LOCIS-generated  reports  are  sufficient  to  support  maintenance 
stand-up  meetings. 

ummmmm 

2 

3 

4 

5 

Strongly  Disagree 

Group  1 

i 

D 

Q 

o 

0 

Group  2 

0 

n 

n 

D 

1 

B.3.1.2  LOCIS  Information  Displays 
B.3.1.2.1  Schedule 

LUG  members  identified  the  importance  of  having  access  to  today's  recap  information. 
Maintainers  track  aircraft  throughout  the  day,  particularly  after  each  flight,  to  stay  up-to-date 
regarding  which  aircraft  are  broken  and  for  what  reason.  LOCIS  must  provide  recap  information 
after  each  go  (set  of  line  numbers),  as  well  as  yesterday’s  recap  information  already  provided 
within  LOCIS.  A  related  issue  is  the  need  for  maintainers  to  access  Debrief  Discrepancies  on  the 
Recap  screen.  LOCIS  must  include  the  reason  for  the  Alpha  landing  code  displayed  on  the  Recap 
screen. 

Users  repeatedly  emphasized  the  importance  of  knowing  which  tail  numbers  are  scheduled  to  fly 
tomorrow.  Two  solutions  were  discussed:  1)  an  identifier  on  the  aircraft  icon  to  indicate  the 
aircraft  as  a  flyer  for  tomorrow;  and  2)  a  “tomorrow  at-a-glance”  view  similar  to  the  existing 
status  at-a-glance  view  depicting  tomorrow’s  activities. 

The  existing  scheduled  maintenance  information  is  not  enough.  Users  want  to  see  what 
maintenance  has  been  scheduled  for  the  week  similar  to  their  “checkerboards” — not  just 
maintenance  due  dates. 

The  following  tables  document  subjective  ratings  for  LOCIS  schedule  displays.  Each  table 
reports  die  number  of  users  assigning  a  given  rating. 


Information  provided  in  LOCIS  schedule  displays  is  I 

Hi RRHHII 

2 
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5 

Not  at  all  Easy  to  Understand 
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Drill-down  information  accessible  from  sortie  (blue)  and 
maintenance  (gray)  activity  blocks  adds  value  to  schedule  data. 


2  3  4  5 

Strongly  Disagree 


2 _ 2  1  0  0 


0  I  4  I  0  I  (HO 


Providing  a  single  “snapshot  in  time”  view  for  the  All  Aircraft  and 
Flyers  displays  improves  their  usabllitv. 


2  3  4  5 

Strongly  Disagree 


0 


0  14  10  I  0  I  0 


The  Scheduled  Maintenance  display  offers  ready  access  to  data  that 
is  currently  difficult  to  consolidate. 

3 

4 

5 

Strongly  Disagree 

2  1 

T" 

T 

0 

0  3 

1  j 

0 

0 

Group  1 _ 2 _ 1 2 0 0 _ 

Group  2 _ 0 _ 3 1 _ 0_  J) _ 

B.3. 1.2.2  Status 

Users  expressed  a  need  for  data  on  NMC  aircraft,  along  with  each  aircraft’s  respective 
discrepancies  and  corresponding  ETICs.  A  potential  solution  is  to  create  a  screen  under  status 
that  includes:  tail  #,  status,  driving  discrepancy,  ETIC,  JCN,  Doc  #,  and  EDD.  A  “nice  to  have” 
capability  would  be  to  link  the  781 A  and  K  write-ups  for  the  selected  tail  number.  The 
“Summary  Discrepancy  Table”  associated  with  the  aircraft  availability  threshold  (as  depicted  in 
the  Spiral  2  demonstration  and  the  LOCIS  Block  Release  version  implemented  at  Hurlburt)  could 
satisfy  this  requirement. 

The  following  tables  document  subjective  ratings  for  LOCIS  status  displays.  Each  table  reports 
the  number  of  users  assigning  a  given  rating. 

The  LOCIS  wing-level  aircraft  status  display  allows  a  user  to 
achieve  a  good  top-level  understanding  of  current  wine  conditions. 


2  3  4  5 

Strongly  Disagree 


0 


2  I  2  I  0  I  0  I  0 


The  level  of  detail  provided  in  the  wing-level  aircraft  status  display 
Is 


5 

Not  at  all  Appropriate 


0 


0 


Information  provided  in  the  geographic  layout  display  is 


2  3  4  5 

Not  at  all  Easy  to  Understand 


0 


0 


Group  2 


Drill-down  information  accessible  from  aircraft  icons  adds  value  to 
aircraft  status  data. 


1  2  3  4  5 

Strongly  Agree  Strongly  Disagree _ 


0 


4  10  10  10  10 


B.3.1.2.3  Munitions 

LUG  members  viewed  the  munitions  capability  (including  accessibility  to  MSC2)  to  be  both 
powerful  and  useful;  however,  they  perceived  it  as  a  whole  to  be  fairly  specialized — i.e.,  one 
likely  to  be  used  by  only  a  few  users  who  have  specific  needs  for  munitions  data  (e.g.,  Pro  Supers 
or  those  with  munitions  expertise).  Munitions  experts  in  the  group  recommended  the  inclusion 
of  additional  capability — namely,  the  integration  of  munitions  training  data  with  the  live 
munitions  data  currently  provided  via  LOCIS,  as  well  as  the  inclusion  of  munitions  status  data 
(FMC  versus  NMC).  Munitions  experts  also  identified  the  usefulness  of  a  new  view  (with  daily 
updates)  to  provide  information  on  projected  expenditures. 

The  following  tables  document  subjective  ratings  for  LOCIS  munitions  displays.  Each  table 
reports  the  number  of  users  assigning  a  given  rating. 


Presenting  munitions  data  in  a  table  format  makes  munitions 
displavs 


5 

Not  at  all  Easy  to  Understand _ 


0 


1  I  2|  1  |  0  |  0 


The  MSC2  analysis  tool  available  as  drill-down  capability  from 
LOCIS  munitions  displays  {Munitions  Detail  hyperlink)  adds  value 
to  munitions  data.. 


5 

Strongly  Disagree 


0 


0 


The  color-coding  provided  in  the  Availability  Forecast  view  is 


2  3  4  5 

Not  at  all  Meaningful 


5  0  0  0  0 


2  I  2  I  0  I  0  I  0 


B.3. 1.2.4  Print  Screen 


LUG  members  believed  the  print  screen  option  to  be  a  good  one.  They  recommended  a  set  of 
appropriate  default  settings  for  each  screen.  For  color-coded  objects,  LUG  members  identified 
the  need  for  some  type  of  redundant  coding  strategy  (to  support  those  using  black  and  white 
printouts).  LUG  members  also  identified  a  need  to  print  a  report  specifying  status,  driving 
discrepancy,  ETIC,  and  JCN  for  all  NMC  aircraft,  suggesting  that  such  a  printout  would  be 
useful  when  used  with  current  EMOC  sheets.  Printouts  of  the  Recap  view  would  be  useful 
during  production  meetings  and  stand  ups,  or  a  (new)  view  containing  data  on  NMC  aircraft. 

The  following  tables  document  subjective  ratings  for  the  LOCIS  print  screen  option.  Each  table 
reports  the  number  of  users  assigning  a  given  rating. 


The  “print  screen”  capability  is  sufficient  to  support  maintenance 
stand-up  meetings. 

1 

Strongly  Agree 

2 

3 

4 

5 

Strongly  Disagree 
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0 

Group  2 

1 
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0 

The  “print  screen” 
maintenance  stand-u 

capability  would  provide  better  support  for 
p  meetings  than  a  report  generator  canabilitv. 
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Strongly  Agree 
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Strongly  Disagree 
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B.3. 1.2.5  Tools 
Forecasting 

LUG  members  were  extremely  favorable  toward  the  forecasting  capability  implemented  in 
LOCK.  They  believed  the  capability  would  be  used  frequently — but  in  different  ways  by 
different  types  of  users.  Pro  supers,  for  example,  would  most  likely  use  the  forecasting  capability 
to  build  various  “rack  and  stack”  lists  (i.e.,  use  it  as  an  analysis  tool).  On  the  other  hand,  most 
senior-level  managers  would  primarily  be  interested  in  the  analysis  products,  using  them  to 
review  or  monitor  current  conditions.  LUG  members  also  believed  the  LOCK  forecasting 
capability  would  save  a  significant  amount  of  time  in  generating  aircraft  rankings  for  different 
types  of  scenarios.  (LUG  members  estimated  that  with  current  methods,  the  kind  of  aircraft 
ranking  provided  by  LOCK  will  typically  require  a  day  to  build.)  LUG  members  suggested 
several  enhancements  to  the  scenario  settings  section  of  the  tool  (e.g.,  changing  the  term  “system 
integrity”  to  “system  reliability”,  allowing  users  to  specify  multiple  MDSs,  and  providing  users  a 
clearer  indication  that  system  integrity/munitions  requirements  and  evaluation  factor  ratings  are 
part  of  the  scenario  setup).  When  considering  the  Health  of  Fleet  (HOF)  Table,  LUG  members 
indicated  that  additional  drill-down  or  rollover  information  accessible  from  System  Integrity  cells 
would  be  useful.  Such  data  would  be  provided  by  tail  number  and  WUC.  Suggested  drill-down 
data  included  discrepancy,  corrective  action,  and  repeat/recur.  Suggested  rollover  information 
for  system  integrity  cells  was  the  set  of  “top  three”  write-ups. 
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The  following  tables  document  subjective  ratings  for  the  forecasting  decision  support  capability 
implemented  in  LOCIS.  Each  table  reports  the  number  of  users  assigning  a  given  rating. 


The  table  format  lends  itself  well  to  the  presentation  of  HOF  data.  | 
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B.3.1.2.6  LOCIS  Notifications 

Alert  and  Warning  Message  Log 

In  assessing  the  Alert  and  Warning  Message  Log,  LUG  members  identified  the  need  for  a  more 
meaningful  description  of  the  events  leading  to  a  change  in  threshold  status.  LUG  members  felt 
the  log’s  usefulness  would  be  improved  if  it  could  help  users  gain  a  clearer  understanding  of 
what  happened  to  cause  status  changes.  Along  these  same  lines,  LUG  members  believed 
threshold  drill-down  information  could  be  made  more  meaningful  if  it  clearly  indicated  the  event 
driving  the  status  change  (and  triggering  an  alert  or  warning).  In  the  current  implementation  of 
LOCTS,  for  example,  when  a  launch  sequence  warning  is  generated  and  the  user  drills  down  for 
further  details,  LOCIS  pushes  the  Launch  Sequence  view.  However,  without  a  “pointer” 
directing  the  user  to  the  problematic  line  number,  that  user  must  spend  additional  time 
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diagnosing  the  problem:  determining  which  line  number  has  triggered  the  warning,  locating  the 
data  for  that  line  number,  and  then  identifying  the  step(s)  with  late  completion  times. 

The  following  tables  document  subjective  ratings  for  the  Alert  and  Warning  Message  Log.  Each 
table  reports  the  number  of  users  assigning  a  given  rating. 
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B.3.1.2.7  Threshold  Settings 

LUG  members  liked  the  threshold  settings  approach  implemented  in  Spiral  3  (i.e.,  linking  each 
threshold  to  an  individual  LOCIS  view  and  accessing/editing  threshold  settings  from  that  view). 
They  also  validated  the  business  rules  implemented  for  the  launch  sequence  threshold.  In 
discussing  appropriateness  of  the  settings  currently  implemented  for  LOCIS’  launch  sequence 
threshold,  LUG  members  confirmed  the  need  for  users  to  select  the  launch  sequence  steps  they 
wish  to  monitor  and  then  specify  the  times  at  which  these  steps  will  be  considered  late.  (The 
user-specified  times  will  serve  as  triggers  for  warnings  and  alerts.)  They  also  recommended  that 
the  option  All  be  included  in  the  drop-down  list  of  line  numbers  and  that  each  line  number  be 
identified  with  its  respective  organization.  When  considering  settings  for  the  aircraft  availability 
threshold,  LUG  members  confirmed  a  need  to  know  the  total  number  of  aircraft  associated  with  a 
selected  MDS. 

The  following  tables  document  subjective  ratings  for  the  threshold  settings  approach.  Each  table 
reports  the  number  of  users  assigning  a  given  rating. 
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B.3.1.2.8  Customization  -  Workspace  Builder 
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LUG  members  believed  the  Workspace  Builder  to  be  a  user  friendly  customization  feature,  and 
they  found  the  set  up  process  easy  and  simple.  LUG  members  liked  the  “drag”  option  for 
resizing  windows.  While  they  also  liked  the  right/left  and  up/down  arrows  that  could  be  used  to 
expand  and  collapse  windows,  LUG  members  preferred  the  more  traditional  “box”  icons  used  in 
windows-based  interfaces.  LUG  members  indicated  the  difficulty  of  remembering  the  name  of 
each  view. 
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Appendix  C  -  LOCIS  Requirements 


Name _ 

Identify  User  _ 


Identify  Job  Responsibility _ 

Identify  Position  and  User  Access  Restrictions 


Identify  Subject  /  Topic _ 

[identify  Default  Data  Elements _ 


Identify  Default  Data  Changed  by  User _ 

Identify  Placement  of  Data _ 

Define  Resources 


Define  User  Thresholds _ 

Define  Formal  Message  Threshold 


Define  SITREP  Threshold 


Define  Disaster  Threshold 


Establish  Available  A/C  Threshold 


Establish  FMC  Rate  Threshold 


[Establish  PMC  Rate  Threshold _ 


[Establish  Utilization  Rate  Threshold 


Establish  Threshold  for  Scheduled  Maintenance  Actions  Canceled 
Establish  Systems  Status  Threshold 


Establish  ETIC  Slips  By  Systems  Threshold 


Establish  Systems  Reliability’s  Threshold 


Establish  Abort  Rate  Threshold 


Establish  Repeat/  Recur  Rate  Threshold 


Define  Munitions  Thresholds 


Define  Supply  Thresholds 


Establish  Threshold  for  Parts/Supplies  for  AGE 


Establish  Threshold  for  Parts/Supplies  Vehicles 


I  Establish  Threshold  for  Parts/Supplies  Facilities 


Set  Minimum  Number  of  On-Hand  Gallons  Threshold 


Set  Maximum  Number  of  Days  to  Obtain  Resupply  Threshold 


Set  Consumption  Rate  Threshold 


Set  Minimum  Number  of  Hot  Pits  Threshold 


| Set  Minimum  Number  of  Panagraphs  Threshold 


Set  Minimum  Number  of  POL  Trucks  Threshold 


Set  Minimum  Number  of  Trained  Drivers  Threshold 


Define  Transportation  Threshold 


Define  AGE  Threshold 


Define  Personnel  Threshold 


Define  Facilities  Threshold 


Define  Threshold  Combinations 


Receive  Weekly  Flying  Schedule  (Peace-Time  Sortie  Generation  & 
Munitions  Requirements)  _ 

Receive  ATO  (War-Time  Sortie  Generation  &  Munitions 
Requirements) 


|  IPEFO  Node  I  Spiral  I 


42 

Sort  Requirements  by  Functional  Area 

43 

Define  Vehicle  Consumption  Standards  Requirements 

44 

Define  POL  Consumption  Standards 

45 

Define  Wing  Aircraft  System  Configuration 

46 

Define  AGE  Standards 

47 

Define  Logistics  Personnel  Skills  Standards 

48 

Project  Future  Events  /  Requirements  , 

49 

Determine  How  Shortfall  Can  Be  Rectified 

[Determine  if  the  Unit  Can  Support  A  Requirement 
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Number 

Name 

IDEFO  Node  Spiral  1 

51 

Identify  Timing  Associated  With  Supporting  A  Requirement 

A3323 

52 

Dispatch  Current  and  Future  Requirements  to  All  Functional  Areas 

A34 

53 

Adjust  Mission  Requirements 

A35 

54 

Determine  Diagnostic  Method 

A41 

55 

Collect  Specified  Threshold  Actual  Data 

A421  1 

56 

Compare  Actuals  to  Thresholds 

iKsin 

Spiral  2 


57 

Evaluate  Combined  Thresholds 

58 

Identify  Negative  Trends  (Timing) 

59 

Identify  NMCS  Status 

A4411 

60 

Identify  PMCS,  PMCM,  PMCB  A/C 

A4412 

61 

Identify  Available  A/C  Location 

lJ. 

62 

Identify  A/C  Configuration 

m\ 

Identify  Available  Types _ 

Identify  Available  Munitions  Location  (On-Site  and  Off-Site  Storage, 

etc.) _ 

Identify  Type  Available  and  Quantity _ 

Identify  Available  Components  (On-hand  or  in  Supply) 


Assign/Identify  Available  Tail  Numbers  To  Mission 

A443 

Assign/Identify  Available  Munitions  to  Mission 

A444 

Determine  Shortfalls  Between  Task  And  Resources 
’  Identify  NMCS/MICAP,  NMCM,  NMCB  A/C 


71 

Identify  A/C  Location 

72 

Identify  Scheduled  Maintenance 

73 

Identify  Real-Time  Sortie  Generations  A/C  Discrepancies  As  They 
Occur 

Identify  A/C  Scheduled  for  Maintenance  Problems 
Identify  Munitions  Handling  Equipment  Problems 


Identify  Site  Survey  Problems 

A4612 

Identify  Parking  Problems 

A46131 

Identify  Re-Fueling  Problems _ 

Identify  ICT  Problems 


Identify  Actual  Beddown  Location  Resource  Shortfalls _ 

Identify  Preliminary  Buildup  Action  Problems 


Identify  Essential  Personnel  Recall  Problems _ 

Identify  LIMFACs  /  Shortfalls 


Identify  Deployed  /  Cross  Country  A/C,  People  and  AGE  Recall 
Problems 


Identify  TPFDD  Tasking  Problems _ 

Identify  Airlift  Schedule  Problems 


Identify  War  Reserve  Material  Problems 


Identify  Redeployment  /  Reconstitution  /  Retrograde  Problems _ 

Identify  Contracting  Problems 


Identify  Communications  Problems _ 

Identify  Security  Problems 


Identify  Weather  Problems 


Identify  Civil  Engineer  /  Airbase  Operability  /  Disaster  Preparedness 
Problems 

Identify  Services  Problems 


Identify  Problems  with  Shift  Assignments _ 

Identify  Problems  with  Training 


Identify  Unavailable  Personnel  Due  To  Legal/Disciplinary  Actions 

Identify  Unavailable  Personnel  Due  To  Pre-Flight  Physicals _ 

Identify  PRP  /  Clearance  Problems 


(identify  Joint  Logistics  System  Problems 


A4641 

A4642 


A46431 

A46432 


A46433 


A4651 
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Name _ 

Identify  Vehicle  Support  /  Maintenance  Problems _ 

Identify  Lean  Logistics  Resupply  /  Transportation  Problems 

Identify  Joint/Coalition  Facility  Problems _ 

Identify  Command  Activities  /  Structures  Problems 


1  Prioritize  Problem _ 

Determine  Sortie  Generation  Command  &  Control  Problem 

Causes/Indicators _ 

Identify  Sources  of  Supply 


Identify  Quantity  On-Hand 


Identify  Parts  Arrival  Time 


Identify  Mode  Of  Shipment _ 

Identify  Location 


Identify  Path  Of  Arrival _ 

Identify  Date  Of  Arrival _ 


Flight  Line  Requisitions  Part _ _ 

Supply  Checks  For  Base  Availability 


Issue  Spares  From  Available  Stock 

MOC  Pushes  Highlighted  A/C  Button  Problem  Spelled  Out 


[LOCIS  Users  See  Options  for  A/C _ 

Pro-Super  Spares  Aircraft  with  New  A 1C _ 

LOCIS  Moves  Original  A/C  into  Mainten an ce  Area 


LOCIS  Moves  New  A/C  into  Original  Flying  Line _ 

Requisition  From  Depot 


Requisition  From  Lateral  Unit 


Expedite  Depot  Repair  Through  Lean  Logistics 


IDEFO  Node  Spiral  1 

A46521 

A46522 _ 

_ A46523 

A 


A48  1 

A52 

A5311  ~ 


A5312  1 


A5313I 


A53132  | 

A53133 


A53134 _ 

A53I35 


A53141 _ 

A53142 


A53143 _ 

A5314411 


A5314412 _ 

A5314413 _ 

A5314414 


A5314415 _ 

A531442 


A531444 


126 

Receive/Expedite  Inbound  Parts  Through  Military/Commercial 
Channels 

A531445 

127 

Auto  Indication  In  MOC  That  No  Spare  Receiver  Is  Available  In 
Supply 

A5315 

128 

Identify  Status  Of  Contracting  Efforts  To  Support  Mission 

129 

Reprioritize  Contracting  Efforts  To  Fill  Critical  Requirements 

A5317 

130 

Identify  Available  POL 

A532 

Identify  Available  AGE _ 


Identify  Personal  Information 


Identify  Assigned  Tasks _ 


Identify  Skill  Level 


Identify  Qualifications _ _ _ 


Identify  Certifications 


I  Select  Qualified  Personnel  _ 

Identify  Authorized  Personnel 


Identify  Types  Of  Needed  Facilities 


Identify  Facilities  Status 


Identify  Facilities  Alternate  Uses _ 

Identify  Maintenance  Status  Of  All  Assigned  Equipment/Circuits 


143 

Monitor  Water/Waste/Power  Requirements 

144 

Identify  Available  Vehicles 

145 

Combine  Problem  Type,  Timing,  &  Available  Resources 

146 

Identify  Alternate  A/C 

147 

Track  Breakout  And  Delivery  Of  Tanks 

148 

Track  Breakout  And  Delivery  Of  Pods 

149 

Track  Breakuul  And  Dclivciy  OfMuuilkm* 

Track  Reprogramming  Of  ECM  Systems 

151 

Track  Uploading  Of  OFP  Updates 

Load  Live  Munitions  IAW  Deployment  Order 
Assign  Mission 


A533 


A53411 


A53412 


A53413I 


A534132 


A534133 


A53414 


A5342 
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A5353 

A5354 


A54221 


A54222 


A54223 


A54224 


A54225 


A54226 


A542311 
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Number 


54  _ 

55 


56  _ 

57 


Name 

A 

Cl 


Check  Next  Mission _ 

Decide  Whether  To  Tum  Or  Repair 


158 

Assign  To  Expediter 

A542331 

159 

Assign  To  Appropriate  Repair  Locations 

A542332 

160 

Re-Prioritize  Solutions 

A54234 

161 

Check  Availability  Of  Technicians  To  Perform  Unscheduled 
Maintenance 

A54241 

162 

Check  Availability  Of  Equipment  To  Support  Unscheduled 
Maintenance 

A54242 

163 

Determine  Per  Cent  Completed 

A542431 

164 

Determine  Engine  Location 

A542432 

_ _ i 

A542322 

A542323 


> 

Move/ Adjust  Internal  (Wing)  Resources 

A5425 

> 

Collect  Info  On  Wing  Shortfalls 

A542611 

Review  Shortfall  Information 

A542612 

Transmit  Info  To  Higher  Headquarters 

A542613 

Convey  Problem  To  Other  Unit 

A542621 

Discuss  Solutions 

A542622 

Request  Assistance _ 

Prioritize  Solution 


Present  Problem  /  Solution 


Report  Completed  Munitions  Load _ 

Report  POL  Status 


Report  Pilot  Show  At  A/C _ 

Report  Maintenance  Actions  Status 


Report  A 1C  Taxi _ 

Report  A/C  Take-off 


Report  A/C  In-Flight  Status _ 

Report  ABDAR  Status 


Report  A/C  Location _ 

Report  POL  Trucks  Dispatched  To  Tum  Spots 


Report  Ammo  Reconciliation _ 

Report  Flight  Crew  Debrief 


Report  Land  Times 


Capture  Actual  Data 


Identify  Level  of  Effort  (Quick  Fix,  etc.) 


A542623 

A543 


m 

Report  Munitions  Arrival  at  Ramp 

■ 

lJ 

Report  Loads  Begin 

m 

m 


3 

4 

5 

1 

6 

1 

A5517 

A5518 


A55211 

A55212 


A55213 

A55361 


A55362 

A55363 


A55364 

A55365 


A55366 


192 

Select  Solution 

193 

Dispense  Changes  to  Impacted  Functional  Areas 
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Appendix  D  -  Reference  Documents 


Document  No. 

CDRL  No. 

Title 

1004240 

N/A 

Program  Plan  of  Execution 

1004241 

N/A 

Software  Development  Plan 

1004242 

N/A 

Risk  Plan 

1004243 

N/A 

Data  Collection  Plan 

1004244 

N/A 

Software  Quality  Program  Plan 

1004245 

A001 

Monthly  Status  Report 

1004246 

N/A 

C/C++  Style  Guide 

1004247 

A008 

Data  Accession  List 

1004248 

N/A 

Operational  Architecture  Specification 

1004249 

A015 

System  Requirements  Specification 

1004250 

N/A 

Java  Style  Guide 

1004251 

A006 

Program  Management  Review  Minutes  Package 

1004252 

A006 

Preliminary  Design  Review  Minutes  Package 

1004253 

N/A 

Year  1  Functional  Scenario 

1004254 

A004 

Contract  Funds  Status  Report 

1004255 

A010 

Demonstration  Plan 

1004256 

A006 

Critical  Design  Review  Minutes  Package 

1004257 

A012 

Training/Administrators  Plan 

1004258 

A014 

Software  Version  Description 

1004259 

A011 

Software  Product  Specification 

1004260 

A016 

Technology  Transition  Report 

1004261 

A009 

Spiral  1  Final  Report 

1004262 

N/A 

Year  2  Operational  Architecture  Specification 

1004263 

A006 

Spiral  2  PMR/PDR  Minutes 

1004264 

A006 

Spiral  2  CDR  Minutes 

1004265 

A015 

Spiral  2  System  Requirements  Specification 

1004266 

N/A 

Block  Release  User's  Manual 

1004267 

N/A 

Block  Release  Training  Guide 

1004268 

N/A 

Spiral  3  System  Document 

86 


Document  No. 

CDRL  No. 

Title 

1004269 

N/A 

Hurlburt  Living  Lab  Sys  Admin  Plan 

1004270 

A006 

Spiral  3  PMR  Minutes 

1004271 

A009 

Spiral  2  Final  Report 

1004272 

A006 

Spiral  3  PDR  Minutes 

1004273 

A016 

Spiral  2  Technology  Transition  Report 

1004274 

A010 

Spiral  2  Demonstration  Plan 

1004275 

A014 

Spiral  2  Software  Version  Description 

1004276 

A011 

Spiral  2  Software  Product  Specification 

1004277 

A006 

Spiral  3  CDR  Minutes 

1004278 

A012 

Spiral  3  Admin/Training  Plan 

1004279 

A010 

Spiral  3  Demonstration  Plan 

1007830 

N/A 

Spiral  3  Operational  Architecture  Specification 

1007831 

N/A 

Spiral  3  Hurlburt  Field  Load,  Installation,  and  Configuration 

1007832 

A009 

Spiral  3  Final  Report 

1007833 

A016 

Spiral  3  Technology  Transition  Report 

1007834 

N/A 

System  Security  Authorization  Agreement 

1007835 

A014 

Spiral  3  Software  Version  Description 

1007837 

A011 

Spiral  3  Software  Product  Specification 

1007838 

A015 

Spiral  3  System  Requirements  Specification 
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1  Abstract 


This  paper  describes  an  initial  effort  to  measure  SA  in  the  context  of  logistics  command 
and  control.  Logistics  Control  and  Information  Support  (LOCIS)  is  a  prototype  decision 
aid  intended  to  allow  a  maintenance  supervisor  to  perform  his/her  job  more  quickly 
without  any  loss  of  SA.  A  literature  review  revealed  that  existing  measures  devised  to 
assess  SA  in  the  context  of  aviation  do  not  transfer  well  to  command  control  tasks. 
Therefore,  existing  measures  of  SA  were  adapted,  and  then  piloted  in  a  preliminary  study 
designed  to  assess  the  impact  of  LOCIS  on  maintenance  supervisor  SA.  Findings 
indicate  that  LOCIS  provides  support  for  building  SA  in  the  context  of  two  specific  tasks 
key  to  the  maintenance  supervisor  job:  preparing  for  the  Daily  Aircraft  Maintenance 
Morning  meeting  and  planning  for  a  deployment.  A  third  task,  providing  recap 
infortnation  at  shift  change,  was  planned  but  adequate  data  was  not  collected  due  to 
technical  difficulties.  Although  these  findings  are  based  on  a  very  small  sample  of  four 
maintainers  —  and  therefore  of  limited  generalizability  —  this  study  represents  a  first  step 
toward  assessing  SA  in  command  and  control  tasks.  Recommendations  for  assessing  SA 
in  a  full-scale  field  study,  as  well  as  recommendations  for  improving  LOCIS  are  included 
in  this  paper. 
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2  Introduction 


Logistics  Command  and  Control  Information  Support  (LOCIS)  is  a  prototype  decision 
aid  designed  to  present  information  to  maintenance  supervisors  in  a  timely  and 
understandable  format.  The  overall  goal  of  this  system  is  to  allow  the  maintenance 
supervisor  to  accomplish  his/her  job  more  quickly  without  a  loss  of  Situation  Awareness 
(SA). 

Maintenance  supervisor  is  a  generic  term  used  in  this  paper  to  describe  the  people  who 
manage  and  supervise  logistics  and  maintenance  for  an  Air  Wing  from  the  non¬ 
commissioned  officer  in  charge  or  officer  in  charge  up  to  the  maintenance  group 
commander.  These  users  comprise  a  team  that  is  tasked  with  ensuring  that  daily  flying 
schedules  are  met,  monitoring  the  scheduled  repair  of  broken  aircraft,  planning  for  future 
deployments  and  TDYs,  noting  trends  in  aircraft  breakage,  and  promoting  seamless 
transition  of  information  between  shifts.  Although  each  has  additional  responsibilities, 
these  are  the  tasks  LOCIS  is  intended  to  support. 

LOCIS  presents,  in  a  compact  and  readable  format,  information  that  is  currently  obtained 
via  disparate  sources  including  various  databases,  support  personnel,  the  monitoring  of 
radio  traffic,  and  meetings  of  different  team  members.  LOCIS  users  are  able  to  obtain  a 
quick  view  of  the  flying  schedule  for  the  day  using  the  “schedule  at  a  glance” 
visualization,  obtain  recap  information  about  previous  flights  using  the  “recap  screen,” 
and  use  the  “health  of  the  fleet  forecasting  tool”  to  identify  the  aircraft  most  suitable  for  a 
specific  deployment  or  TDY.  Additionally,  information  about  the  status  of  broken 
aircraft  is  available  via  a  number  of  paths  within  LOCIS.  The  intent  of  LOCIS  is  to 
provide  maintenance  supervisors  information  critical  to  maintaining  accurate  SA  in  an 
easily  accessible  format  that  is  at  their  fingertips. 

This  paper  describes  a  pilot  study  conducted  to  assess  the  feasibility  of  measuring  the 
impact  of  LOCIS  on  maintenance  supervisor  SA.  Existing  methods  for  measuring  SA 
were  developed  in  the  context  of  military  aviation.  Not  surprisingly,  measures  developed 
to  assess  S  A  in  the  aviation  domain  do  not  generalize  directly  to  command  and  control 
environments,  with  tasks  involving  a  longer  timeline,  an  ill-defined  mission  beginning 
and  end,  and  a  strong  monitoring  component. 

Due  to  the  fact  that  a  great  deal  of  SA  testing  has  dealt  with  actual  aircraft  “pilots”  as 
subjects,  from  this  point  on  this  document  will  refer  to  our  “pilot”  study  as  a  preliminary 
study  (to  reduce  terminology  confusion). 

A  small  preliminary  study  was  conducted  in  preparation  for  a  larger  scale  study  to  assess 
S  A  in  maintenance  supervisors.  The  study  was  constrained  by  a  number  of  factors, 
including  a  limited  number  of  SMEs  available  to  participate  in  the  study,  the  lack  of  a 
real-time  simulation,  and  limited  time  with  each  participant.  Working  within  these 
constraints  we  reviewed  literature  about  existing  SA  measurement  techniques,  developed 
an  adapted  methodology,  and  tested  it  on  four  experienced  maintenance  supervisors. 
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The  situation  awareness  history  and  definition  section  of  this  paper  briefly  describes 
the  history  and  definition  of  SA  in  aviation;  contrasts  the  characteristics  of  the  piloting 
task,  the  traditional  area  of  study  for  SA,  with  the  maintenance  supervisor  task;  and 
briefly  describes  existing  measures  of  SA.  The  preliminary  study  section  describes  the 
adapted  methodology  in  the  context  of  the  preliminary  study.  The  lessons  learned  about 
measuring  SA  section  describe  lessons  learned  with  a  view  toward  a  full-scale  field 
study  assessing  the  impact  of  LOCIS  on  maintenance  supervisor  SA.  The  final 
conclusions  section  summarizes  our  conclusions  regarding  measuring  SA  and  highlights 
our  recommendations  for  improving  LOCIS.  ° 

3  Situation  awareness  history  and  definition 


3.1  History 

The  term  SA  came  into  common  use  in  the  context  of  World  War  I.  In  Spick’s  1998,  The 
Ace  Factor,  SA  is  described  as,  .  .a  combination  of  many  things,  but  in  essence  it  is  the 
ability  of  the  pilot  to  keep  track  of  events  and  foresee  occurrences  in  the  fast-moving 
dynamic  scenario  of  air  warfare”  (p.  vi).  Pilots  have  found  the  term  to  be  useful  in 
describing  aspects  of  the  fast-paced,  high-risk  task  of  piloting  an  aircraft  as  they  have 
learned  from  their  own  experiences  and  from  each  other.  Successful  pilots  were  able  to 
quickly  perceive  important  environmental  cues,  understand  their  significance,  and 
accurately  predict  the  dynamics  of  the  situation  minutes  into  the  future  so  that  they  could 
be  prepared  to  react. 


Over  the  years,  situation  awareness  has  proven  to  be  powerful  concept  for  describing  the 
state  pilots  are  trying  to  achieve.  However,  it  wasn’t  until  the  1980s  that  researchers 
began  to  emphasize  SA,  both  in  terms  of  basic  research  to  understand  the  concept  and 
applied  research  to  develop  improved  training  and  system  design.  The  past  20  years  has 
seen  an  explosion  in  training  intended  to  prepare  pilots  to  achieve  situation  awareness  in 
time-pressured,  high-risk,  and  sometimes  complex  situations,  as  well  as  system  and 
display  design  strategies  aimed  at  aided  pilots  in  attaining  good  situation  awareness. 

3.2  Definition 

Although  a  widely  agreed  upon  definition  of  SA  has  proven  elusive  (Dominguez,  1994; 
Jeannot,  2000),  Endsley  (1988)  offers  the  following;  “the  perception  of  the  elements  in 
the  environment  within  a  volume  of  time  and  space,  the  comprehension  of  their  meaning 
and  the  projection  of  their  status  in  the  near  future.”  Researchers  do  agree  that  three  key 
elements  of  SA  include:  1)  perceiving  relevant  cues,  2)  comprehending  or  interpreting  the 
cues,  3)  and  projecting  an  understanding  of  the  situation  to  forecast  future  situation 
events  and  dynamics. 

However,  in  studying  the  maintenance  supervisor  task,  it  is  important  to  take  into  account 
the  concept  of  team  SA,  in  addition  to  the  individual’s  SA.  The  maintenance  supervisor’s 
SA  is  dependent  not  only  on  his/her  own  knowledge  of  flightline  activities,  but  on  the 
information  shared  with  members  of  his/her  team.  Therefore,  maintaining  a  shared 
understanding  of  the  situation  at  hand  is  important  to  successful  performance  for  each 
team  member,  especially  the  maintenance  supervisor.  Endsley  (1995)  specifies  that  team 
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SA  consists  of  the  SA  required  of  each  team  member  and  the  overlapping  S  A  that  occurs 
between  and  among  team  members,  especially  for  coordination.  Building  on  her 
definition,  it  follows  that  some  members  of  the  team  need  a  great  deal  of  overlapping  SA 
for  successful  performance.  For  example,  the  night  shift  maintenance  supervisor  and  the 
day  shift  maintenance  supervisor  have  mechanisms  in  place  (i.e.,  reports,  emails,  recap 
discussions)  to  ensure  that  they  have  a  shared  understanding  of  the  situation.  Other 
members  of  the  maintenance  team  such  as  the  avionics  specialist  and  engine  mechanic 
will  have  a  much  smaller  amount  of  overlap  in  their  SA  because  their  roles  and  therefore 
their  focus  of  attention  is  somewhat  different  (see  Figure  1). 


Engine 

Mechanic 


Figure  1.  Notional  Team  Situation  Awareness  (Adapted  from  Endsley,  M.R.  (1995). 
Toward  a  theory  of  situation  awareness  in  dynamic  systems.  Human  Factors,  57(1),  32- 
64. 


3.3  Contrast:  piloting  and  maintenance  supervising 

Recently,  researchers  have  begun  to  generalize  the  construct  of  SA  to  domains  quite 
different  from  aviation  including  anesthesiology  (Gaba,  Howard,  and  Small,  1995),  air 
traffic  control  (Sollengerger  &  Stein,  1995),  and  nuclear  power  plant  operation  (Collier  & 
Folleso,  1995)  (p.  278).  As  we  began  to  explore  the  construct  of  SA  in  the  context  of 
aircraft  maintenance,  some  important  differences  between  piloting  and  maintaining 
emerged. 

In  general,  maintenance  supervisors  are  working  on  a  timeline  much  longer  than  that  of  a 
fighter  pilot  and  much  of  their  feedback  does  not  come  as  a  result  of  direct  actions  taken 
on  the  environment.  A3  a  result,  feedback  from  the  environment  may  be  delayed 
considerably  for  a  maintainer  as  compared  to  a  pilot.  Early  in  the  morning  a  maintainer 
may  schedule  a  specific  aircraft  to  be  fixed  by  1600,  in  time  for  the  afternoon  flight. 
However,  at  1500,  s/he  may  find  that  additional  problems  were  discovered  during  the 
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repairs,  and  therefore  the  aircraft  may  not  be  available  to  fly  that  afternoon.  Feedback 
requiring  deviation  from  a  plan  can  take  hours  -  or  even  several  days  -  from  the  time  the 
plan  was  conceived.  Additionally,  feedback  from  the  environment  may  affect  multiple 
plans  being  tracked  by  the  maintenance  supervisor.  For  example,  maintenance  performed 
on  one  aircraft  may  affect  today’s  flying  schedule  as  well  as  plans  for  a  scheduled 
deployment  for  that  aircraft  in  several  weeks.  While  a  pilot  is  often  able  to  take  an  action 
and  get  direct,  immediate  feedback  via  his/her  own  senses,  the  maintenance  supervisor  is 
directing  others  to  take  action  and  receiving  feedback  via  report  and  observation 
mechanisms. 

For  aircraft  maintenance  supervisors,  much  of  the  information  needed  is  not  at  their 
fingertips;  therefore  they  must  engage  in  deliberate  information  seeking  activities. 
Maintenance  supervisors  are  not  relying  on  their  own  perceptual  skills  (i.e.,  what  is 
visible  outside  the  cockpit,  the  sound  of  the  engines,  etc.),  but  on  information  provided  by 
front-line  maintainers.  The  maintenance  supervisor  may  receive  notice  that  a  specific 
aircraft  has  a  fuel  leak  and  is  scheduled  for  repair  at  1600,  but  that  is  not  enough 
information  for  him/her  to  do  his  job  well.  The  maintenance  supervisor  must  talk  to 
others  to  find  out  the  extent  of  the  damage  and  whether  repair  of  the  aircraft  at  1600  can 
realistically  be  expected.  This  may  involve  tracking  down  information  about  supply  (i.e., 
Are  the  needed  parts  available?  If  not,  will  the  parts  be  delivered  in  time?  If  not,  does  it 
make  sense  to  cannibalize  the  parts  from  another  jet?),  information  about  staffing,  and 
information  about  support  equipment  (i.e.,  if  there  are  two  jets  with  fuel  problems  and 
only  one  fuel  bam,  which  jet  gets  precedence?).  Further,  if  the  part  cannot  be  obtained  in 
an  acceptable  time  frame,  the  maintenance  supervisor  will  need  to  collect  information 
about  other  taskings  scheduled  for  the  aircraft  in  order  to  anticipate  the  impact  this  delay 
in  repair  will  have  on  future  activities. 

For  aircraft  maintenance  supervisors,  SA  is  comprised  of  a  more  complex  monitoring 
task  than  the  task  of  a  pilot.  Although  monitoring  the  environment  is  key  to  successful 
piloting,  monitoring  plays  an  even  larger  role  in  the  maintenance  supervisor  task,  and 
encompasses  a  large  number  of  variable  data  sources.  The  maintenance  supervisor 
monitors  the  status  and  activity  of  24  to  72  aircraft  for  today,  tomorrow,  and  as  much  as 
90  days  out  when  preparing  for  a  deployment.  These  activities  tend  to  be  interrelated  in 
that  today’s  activities  directly  impact  tomorrow’s  flights  as  well  as  deployments  planned 
for  the  future.  The  maintenance  supervisor  must  continually  make  trade-offs  and 
maintain  balances  for  each  aircraft  on  a  number  of  parameters.  For  example,  if  an 
aircraft  cannot  fly  the  scheduled  mission  today  or  tomorrow,  overall  flying  horns  for  that 
aircraft  are  decreased  therefore  pushing  back  inspection  schedules  that  are  based  on 
number  of  flying  hours.  Many  of  these  scheduled  inspections  are  mandatory  for 
deployments,  therefore,  the  deployment  plan  may  be  adversely  affected  by  the  plane’s 
inability  to  fly  today  and  tomorrow.  Maintenance  supervisors  are  constantly  tracking  the 
progress  of  routine  maintenance,  unexpected  breaks,  and  upgrades  and  modifications,  as 
well  as  daily  flying  activity  for  each  aircraft. 

For  a  pilot,  the  mission  has  a  clearly  defined  beginning  and  end.  For  maintainers,  tasks 
are  much  more  cyclical  without  a  distinct  beginning  and  end.  Each  of  the  24  to  72 
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aircraft  for  which  the  maintenance  supervisor  is  responsible  is  constantly  moving  through 
a  six-stage  process:  recover,  repair,  fiiel,  arm,  configure,  and  launch.  Aircraft  move 
through  these  stages  at  different  rates  and  many  aspects  of  the  timeline  are  unpredictable. 
Indeed,  unexpected  and  unplanned  repairs  are  a  routine  part  of  the  maintenance  world.  In 
addition,  it  is  not  uncommon  for  a  repair  to  appear  to  be  complete,  only  to  be  discovered 
later  that  the  original  problem  has  not  been  completely  resolved.  Cycles  are  not  limited 
to  the  six  stage  process  for  an  individual  aircraft;  scheduled  deployments  are  dependent 
on  the  interrelationship  among  aircraft  and  schedules.  Lists  of  aircraft  scheduled  for  a 
deployment  are  continually  revised  until  the  time  of  the  deployment.  In  other  words,  the 
list  of  specific  aircraft  tail  numbers  selected  for  a  deployment  may  change  multiple  times 
before  die  deployment.  These  changes  are  all  dependent  on  the  unpredictable  nature  of 
the  processes  occurring  daily  on  the  flightline.  Tlie  maintenance  supervisor  may  add  and 
remove  a  tail  number  from  the  list  several  times  depending  on  other  maintenance 
activities. 

3.4  Measuring  SA 

In  this  section,  we  briefly  review  established  methods  of  measure  SA.  Several  excellent 
reviews  of  the  literature  exist.  See  Jeannot,  (2000),  Perla,  et  al  (2000),  and  Uhlarik  & 
Comerford  (2002)  for  more  thorough  reviews  of  the  SA  literature. 

The  Situation  Awareness  Rating  Technique  (SART)  (Taylor,  1990)  and  Cognitive 
Compatibility  -  Situation  Awareness  Rating  Technique  (CC-SART)  (Jeannot,  2000; 
Taylor,  1995)  are  perhaps  the  most  well  known  tools  for  measuring  SA.  With  these 
methods,  study  participants  provide  self-ratings  on  a  set  of  ten  generic  SA  constructs. 

The  constructs  are  grouped  into  three  categories:  Demand  on  attentional  resources, 
Supply  of  attentional  resources,  and  Understanding.  Ratings  for  all  the  dimensions  are 
combined  to  get  an  overall  SA  value.  These  tools  have  been  used  successfully  in  the 
context  of  decision  tasks  by  navigators  and  pilots  (Selcon  &  Taylor,  1990),  as  well  as  in 
an  air-to-ground  attack  simulation  (Vidulich,  Crabtree,  &  McCoy,  1993). 

SART  and  CC-SART  work  best  as  a  post-task  or  post  real-time  situation  measure. 
Participants  are  asked  to  reflect  on  recent  experience  and  make  ratings  for  each 
dimension.  We  found  this  strategy  less  relevant  for  our  preliminary  test  because  the 
maintenance  task  would  not  be  performed  real-time.  Even  if  we  had  had  a  real-time 
simulation,  maintenance  supervisors  often  do  not  get  clear  feedback  immediately. 
Therefore,  flaws  in  understanding  may  not  be  clear  to  an  individual  for  some  time.  As  a 
result,  we  determined  that  this  type  of  self-rating  would  not  be  feasible  for  command  and 
control  tasks  such  as  maintenance  supervising  which  are  characterized  by  delayed 
feedback  and  multi-hour  or  multi-day  timelines. 

The  Situation  Awareness  -  Subjective  Workload  Dominance  (SA-SWORD)  technique  is 
based  on  the  Analytic  Hierarchy  Process  (Vidulich  &  Hughes,  1991).  This  method 
involves  pairwise  comparison  for  each  experimental  condition.  This  method  was 
developed  in  the  context  of  studies  designed  to  assess  the  efficacy  of  two  or  more 
designs,  hence  the  emphasis  on  pairwise  comparisons.  For  our  study,  the  lack  of 
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parallelism  between  the  use  of  LOCIS  and  the  traditional  means  of  performing  the 
maintenance  supervisor  tasks  made  creating  meaningful  pairs  impractical. 

SA  Rating  Scales  (SARS)  was  developed  specifically  to  measure  SA  in  fighter  pilots 
(Bell  &  Waag,  1995;  Waag  &  Houck,  1994).  Thirty-one  behavior  elements  were 
identified  that  are  considered  important  for  mission  success.  Pilots  were  asked  to  rate 
themselves  and  other  pilots  in  terms  of  S A  on  a  6-point  scale,  ranging  from  acceptable  to 
outstanding.  For  this  line  of  research,  SA  was  regarded  as  an  innate  ability  rather  than  a 
changeable  state  of  knowledge.  Although  SARS  was  found  to  be  very  useful  for 
assessing  SA  in  F-15  pilots  in  the  context  of  training  simulations,  the  definition  of  SA  as 
an  innate  ability  made  application  of  the  SARS  strategy  less  relevant  for  our  study. 

The  Situation  Awareness  Global  Assessment  Technique  (SAGAT)  was  developed  by 
Endsley  (1988).  Participates  are  asked  to  work  through  a  simulation.  At  specific  points, 
the  simulation  is  frozen  and  a  set  of  randomly  selected  questions  is  administered.  All 
data  from  the  simulation  is  removed  while  the  participant  responds  to  the  questions.  The 
accuracy  of  the  participant’s  responses  is  used  as  a  measure  of  SA.  This  technique 
requires  a  real-time  simulation  that  can  be  frozen,  and  therefore  was  not  feasible  for  our 
study.  Furthermore,  this  method  is  best  suited  for  a  task  that  requires  memorization  as  it 
seems  likely  that  participants  in  many  tasks  may  have  accurate  SA,  but  be  unable  to  recite 
the  details  of  the  data  without  referring  to  charts  or  decision  aids  available  on  the  job. 

The  Situation  Present  Assessment  Method  (SPAM)  (Durso,  et  al.,  1995)  makes  use  of  a 
confederate  SME.  Several  times  during  the  task  or  real-time  simulation,  a  realistic 
confederate  calls  and  asks  the  participant  a  question.  Investigators  measure  how  long  it 
takes  the  participant  to  gather  the  requested  information  as  well  as  the  accuracy  of  the 
answer.  The  ability  of  the  confederate  to  present  him/herself  as  a  believable  coworker  is 
critical,  so  that  the  participant  is  unable  to  discern  test  questions  from  other  genuine 
questions  asked  during  the  task.  In  addition  to  a  realistic  confederate,  this  method 
requires  a  high-fidelity  simulation  and  a  task  that  allows  for  potential  interruptions  by  a 
confederate.  These  requirements  eliminated  SPAM  as  a  potential  tool  for  our  preliminary 
study. 

Our  strategy  for  leveraging  previous  work  on  SA  to  adapt  existing  methodologies  for  use 
in  less  dynamic  tasks  is  described  below. 

4  Preliminary  Study 


4.1  Introduction 

Given  the  constraints  of  our  study,  none  of  the  well-established  SA  measures  could  be 
applied  directly.  Therefore,  we  went  back  to  the  definition  of  SA  supplied  by  Endsley 
(1988)  and  developed  a  strategy  that  would  be  feasible  for  measuring  SA  in  a  command 
and  control  task  characterized  by  a  complex  monitoring  component,  a  timeline  that  is 
hours  or  days  rather  than  minutes,  and  feedback  which  is  often  neither  direct  nor 
immediate. 
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In  keeping  with  the  basic  principles  of  SA  measurement  (Endsley  &  Garland,  2000),  We 
determined  that  we  wanted  to  measure  whether  participants  in  our  study  were: 

A)  noticing  the  same  cues, 

B)  understanding  the  cues  in  the  same  way,  and 

C)  forecasting  events  in  the  future  in  a  similar  manner. 

Although  a  realistic  prototype  of  the  LOCIS  decision  support  tool  was  available, 
technical  limitations  prevented  us  from  using  real-time  data  in  this  preliminary  study.  In 
addition  to  technical  limitations,  it  would  not  have  been  feasible  to  conduct  the  study  in 
real  time,  as  SA  unfolds  over  the  course  of  an  8-hour  shift  (at  a  minimum)  for 
maintenance  supervisors.  Therefore,  we  worked  with  software  developers  to  enter  data 
that  would  represent  the  state  of  the  Aircraft  Maintenance  Unit  (AMU)  at  three  different 
points  in  the  day.  A  scenario  was  built  around  each  of  those  points.  Each  scenario 
focused  on  a  specific  task  key  to  the  maintenance  supervisor’s  job.  In  order  to  examine 
situations  in  which  SA  spans  more  than  an  8-hour  shift,  one  scenario  was  designed  in 
which  the  maintenance  supervisor  was  asked  to  plan  an  activity  that  would  not  occur  for 
four  weeks. 

4.2  Methods 

Four  people  from  the  maintenance  community  participated  in  this  preliminary  study:  one 
Technical  Sergeant,  two  Senior  Master  Sergeants,  and  one  Chief  Master  Sergeant.  Each 
of  the  four  Non  Commissioned  Officers  had  over  20  years  experience  in  Air  Force 
aircraft  maintenance.  One  of  the  participants  was  from  the  Air  Force  Special  Operations 
Command  (AFSOC),  and  the  other  three  were  from  Air  Combat  Command  (ACC).  All 
four  participants  are  currently  working  in  maintenance  management  positions. 

The  maintainers  in  this  study  each  participated  in  a  two-hour  session,  beginning  with 
introductions  followed  by  SA  testing.  S  A  testing  consisted  of  the  presentation  of  a 
scenario  in  which  the  participant  was  asked  to  complete  a  task.  During  the  task, 
participants  were  asked  to  think  aloud,  highlighting  information  they  were  seeking  and 
the  significance  of  the  information.  After  each  task,  a  short  interview  with  questions 
directed  at  eliciting  cues  noticed,  an  understanding  of  the  situation,  and  forecasting 
information  was  administered. 

Each  of  the  three  scenarios  targeted  a  task  in  which  SA  in  particularly  important  for 
successful  job  performance  by  the  maintenance  supervisor.  Each  is  described  below. 

Scenario  1:  morning  meeting  preparation.  The  first  scenario  revolved  around 
preparation  for  the  Daily  Aircraft  Maintenance  Morning  meeting.  Each  morning 
the  maintenance  supervisor  meets  with  production  superintendents  and  other 
sortie  production  supervisors  to  update  the  rest  of  the  team  on  the  current  situation 
and  goals  for  the  day,  assign  tasks,  and  obtain  information  to  fill  out  his/her  SA. 
Morning  meeting  preparation  includes  assessing  the  status  of  the  aircraft, 
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reviewing  the  schedule  for  the  day,  prioritizing  actions  for  the  day,  and 
identifying  information  needs  (i.e.,  the  need  to  talk  to  the  maintenance  officer  to 
find  out  why  one  of  those  broken  jets  has  not  been  cannibalized  for  parts.). 
Participants  were  asked  to  prepare  for  the  morning  meeting  using  LOCIS. 

Scenario  2:  deployment  planning.  In  the  second  scenario,  participants  were 
informed  that  orders  had  come  down  asking  them  to  prepare  for  a  four-week 
deployment,  to  take  place  four  weeks  from  today.  They  were  provided 
information  about  the  number  of  aircraft  required,  the  expected  number  of  sorties, 
estimated  flight  time  to  location,  munitions  requirements,  etc.  Using  this 
information  and  interacting  with  LOCIS,  participants  were  asked  to  develop  a 
“rack  and  stack,”  a  document  identifying  the  aircraft  most  suitable  for  the 
deployment  and  listing  all  the  maintenance  activities  that  need  to  take  place  on 
each  aircraft  before  the  deployment. 

Scenario  3:  shift  change  recap.  The  last  scenario  moved  forward  in  time  to  the 
end  of  the  day.  Participants  were  informed  that  the  swing  shift  maintenance 
supervisor  was  new  to  the  unit.  Therefore,  the  participant  would  want  to  conduct 
a  very  thorough  recap  and  highlight  important  issues  that  might  be  overlooked  or 
might  be  potentially  problematic  for  someone  new  to  the  unit.  Participants  were 
asked  to  use  LOCIS  to  pull  together  the  information  they  would  want  to  pass 
along  to  the  swing  shift  replacement. 

After  being  introduced  to  each  scenario,  participants  were  asked  to  think  aloud  as  they 
interacted  with  LOCIS  if  that  was  comfortable  to  them,  and  to  inform  the  interviewer 
when  they  had  completed  the  task.  Because  participants  had  received  limited  training  on 
LOCIS,  they  were  encouraged  to  inform  the  interviewer  if  they  were  looking  for 
information  that  they  could  not  remember  how  to  access  in  LOCIS.  All  participants  were 
able  to  complete  each  task  in  less  than  20  minutes. 

After  completing  each  task,  participants  were  asked  a  series  of  questions  aimed  at 
unpacking  their  situation  awareness.  The  question  series  was  designed  to  elicit 
information  regarding  what  participants  noticed  and  understood,  and  how  they  would 
forecast  the  situation  into  the  future.  These  questions  included: 

•  What  are  the  important  things  you  want  to  discuss  in  the  morning 
meeting/communicate  to  your  swing  shift  replacement?  What  are  the  things 
on  the  rack  and  stack  you  are  paying  particular  attention  to? 

•  What  3  things  are  you  most  worried  about? 

•  What  are  you  priorities  at  this  point  in  time? 

•  What  information  do  you  need  that  you  don’t  have? 

•  What  will  you  do  next? 

•  What  problems  are  you  anticipating? 

A  second  set  of  questions  focused  on  how  the  tasks  included  in  the  scenarios  (i.e., 
morning  meeting,  deployment  planning,  recap)  happen  currently,  without  the  use  of 
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LOCIS.  This  second  set  of  questions  was  aimed  at  helping  us  understand  how  good  SA 
is  achieved  in  the  context  of  current  operations  without  LOCIS.  These  questions 
included: 

•  When  was  the  last  time  you  led  a  morning  meeting/created  a  rack  and 
stack/had  a  recap  discussion  at  shift  change? 

•  Can  you  walk  me  through  how  you  prepared? 

•  Was  the  incident  you  just  described  pretty  typical? 

•  Is  there  information  you  have  access  to  in  your  unit  that  is  not  available  on 
LOCIS? 

•  Is  there  information  in  LOCIS  that  you  do  not  have  easy  access  to  in  your 
unit? 


4.3  Analysis 

Data  from  each  scenario  was  analyzed  to  explore  whether  participants  noticed  elements 
that  scenario  designers  intended  them  to,  understood  the  significance  of  the  data,  and 
forecasted  similar  outcomes  in  the  context  of  completing  each  task. 

Analysis  for  this  preliminary  study  focused  on  examining  these  SA  components  in  two 
ways.  First,  specific  cues  were  included  in  each  scenario.  Our  intent  was  to  determine 
whether  participants  noted  those  specific  cues,  understood  them  in  the  way  intended,  and 
forecasted  them  as  expected.  Information  gathered  from  maintenance  supervisors  was 
compared  with  these  expected  results  to  validate  the  scenario  as  well  as  to  affirm  SA 
provided  by  LOCIS. 

Second,  information  gathered  from  the  four  maintenance  supervisors  was  compared  and 
contrasted  to  more  fully  understand  the  SA  issues  and  implications  associated  with  the 
scenario.  Three  researchers  each  took  the  data  from  one  of  the  three  scenarios  to  be 
analyzed  in  depth.  Each  researcher  restructured  the  interview  notes  to  facilitate 
observation  across  participant  responses  for  an  individual  task  (i.e.,  morning  meeting, 
deployment  planning,  or  end-of-shift  recap).  Each  researcher  examined  the  data  to 
answer  the  following  five  questions,  working  from  a  shared  analysis  guide. 

1.  Did  all  four  participants  notice  the  same  things?  What  did  they  notice?  Provide 
examples. 

2.  Did  all  four  participants  interpret  information  in  the  same  way?  How  did  they 
understand  the  situation?  Provide  examples. 

3.  Did  all  four  participants  predict  future  events  similarly?  What  did  they  forecast? 
Provide  examples. 

4.  In  what  ways  did  LOCIS  support  SA  for  this  task? 

5.  Did  LOCIS  obstruct  or  hinder  SA  in  any  way?  If  yes,  do  we  have  any  candidate 
solutions? 

After  independently  analyzing  the  data,  the  three  researchers  met  with  two  SMES  who 
had  not  participated  in  the  study  to  calibrate  and  discuss  the  findings.  In  addition,  lessons 
learned  and  strategies  for  improving  this  method  of  measuring  SA  were  discussed. 
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4.4  Findings 

4.4.1  Scenario  1:  morning  meeting  preparation 


4.4. 1.1  S A  findings 

Each  participant  in  our  preliminary  study  quickly  established  accurate  SA  during  the  first 
scenario,  noting  the  important  elements  designed  into  the  scenario  and  understanding  the 
implications  of  these  elements  for  meeting  the  day’s  flying  schedule.  All  participants 
noticed  key  elements  of  the  morning  meeting  preparation  scenario  including  a  broken 
flyer  and  a  broken  spare  (backup  aircraft).  All  four  participants  understood  that  this 
situation  was  risky  in  that  if  anything  else  broke  before  the  first  flight,  they  would  not  be 
able  to  make  all  the  scheduled  sorties.  With  regards  to  forecasting,  the  three  participants 
with  the  most  relevant  work  experience  all  suggested  proactive  strategies,  including 
making  phone  calls  and  providing  additional  direction  to  other  team  members  to  ensure 
that  every  effort  was  made  to  fix  the  broken  aircraft,  and  contacting  operations  to  discuss 
the  prioritization  of  flights  in  the  event  that  not  enough  aircraft  would  be  available  to 
meet  the  schedule. 

There  was  more  variability  when  it  came  to  the  level  of  detail  at  which  participants 
investigated  the  larger  issues  described  above.  Some  participants  articulated  more  details 
they  noticed,  such  as  “none  of  the  flyers  are  MICAP  for  parts”1  or  “two  aircraft  are  off 
station.”  Some  noted  flaws  in  the  scenario,  such  as  “A8521  is  PMCS2,  but  should  be 
NMCS3  because  it  has  1 A  MICAP”  or  “this  aircraft  has  a  weight  and  balance  inspection, 
but  it  is  red  -  should  not  be  NMC  for  scheduled  inspection.” 

Furthermore,  some  expanded  on  their  understanding  of  the  situation  beyond  the  key 
elements  related  to  make  the  day’s  flying  schedule.  Some  interpreted  the  fact  that  there 
were  three  aircraft  broken  for  parts  as  an  indicator  that  something  was  wrong  -  or  that  the 
situation  required  additional  investigation.  Generally,  when  a  number  of  jets  are  missing 
parts,  common  practice  is  to  cannibalized  one  jet  to  fix  the  others  until  the  parts  on  order 
are  received.  The  fact  that  this  had  not  happened  was  a  red  flag  to  some.  Some  noticed 
that  tail  numbers  had  been  swapped  the  day  before  and  interpreted  that  to  mean  that  a 
spare  was  flown.  This  was  considered  another  area  for  further  investigation.  Participants 
were  curious  about  the  reason  for  the  tail  swap,  as  that  might  have  implications  for  the 
future  performance  of  the  jet  that  was  not  flown. 

It  is  not  clear  whether  the  variability  in  the  level  of  detail  is  reflective  of  variability  in  SA, 
or  whether  this  is  an  artifact  of  our  methods.  In  future  studies  it  may  make  sense  to  ask 
more  directed  questions  to  assess  whether  participants  are  noticing,  understanding,  and 
forecasting  important  elements  of  the  scenario. 


'2  MICAP  refers  to  mission  essential  parts  that  are  not  readily  available 
PMCS  refers  to  aircraft  that  are  only  partially  mission  capable,  they  can  be  flown  but  will  not  be  able  to 
complete  the  mission  as  defined. 

NMCS  refers  to  aircraft  that  are  not  mission  capable  and  cannot  be  flown 
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4.4. 1.2 


LOCIS  findings 


In  terms  of  supporting  SA,  our  data  indicate  that  LOCIS  will  provide  valuable 
information  to  maintenance  supervisors  in  building  accurate  SA  as  they  prepare  for  the 
morning  meeting.  The  Status  at  a  Glance  feature  provides  a  concise  overview  and  calls 
the  user’s  attention  to  broken  aircraft,  particularly  those  scheduled  to  fly.  All  the 
participants  in  our  study  were  able  to  quickly  identify  which  aircraft  were  scheduled  to  be 
flown,  the  broken  flyer,  the  broken  spare,  and  other  broken  aircraft.  It  was  immediately 
clear  to  all  participants  that  it  would  be  a  challenge  to  meet  the  day’s  flying  schedule. 

This  information  would  traditionally  be  available  via  discussion  with  the  maintenance 
supervisor  from  the  previous  shift  and/or  a  review  of  emails  and  other  documents  created 
on  the  previous  shift.  The  fact  that  LOCIS  is  able  to  provide  this  same  information  real 
time  by  pulling  information  from  existing  databases  represents  a  significant  savings  in 
labor  as  the  Status  at  a  glance  eliminates  the  need  for  other  personnel  to  compile, 
integrate,  and  document  this  information. 

Participants  were  also  able  to  fill  out  their  SA  by  obtaining  detailed  information  about  the 
status  of  each  aircraft.  Participants  used  a  number  of  different  routes  through  the 
software  to  obtain  information  about  the  reason  each  broken  jet  was  not  available,  which 
were  waiting  for  parts,  which  parts  deliveries  were  overdue,  etc.  There  were  also  able  to 
use  the  recap  data  found  in  LOCIS  to  obtain  information  about  the  status  and  history  of 
specific  jets.  LOCIS  also  provided  information  about  inspections  that  were  scheduled  for 
the  current  day,  as  well  as  the  rest  of  the  week  to  aid  in  forecasting. 

Although  LOCIS  in  its  current  form  provided  an  impressive  tool  for  building  SA,  this 
preliminary  study  did  highlight  a  few  types  of  information  maintenance  supervisors 
require  that  were  not  found  in  the  LOCIS  prototype.  Perhaps  most  importantly, 
maintenance  supervisors  reported  that  they  rely  on  a  printout  of  all  broken  aircraft  and 
information  about  the  status  of  each  (reason  for  break,  scheduled  repairs,  etc.)  The 
printout  is  carried  to  meetings  throughout  the  day  and  updated  by  hand.  Maintenance 
supervisors  rely  on  this  printout  to  maintain  up-to-the  minute,  accurate  SA  throughout  the 
day  when  they  are  away  from  the  computer.  It  will  not  be  enough  for  LOCIS  to  provide 
accurate  SA  when  the  user  is  at  the  computer.  Key  printouts  or  mobile  computing 
displays  showing  status  information  must  be  available,  as  a  large  percentage  of  the 
maintenance  supervisor’s  day  is  spent  away  from  the  computer. 

Other  information  needs  reported  by  study  participants  include  the  flying  schedule  for 
tomorrow.  Although  the  capability  to  display  the  flying  schedule  for  the  following  day  is 
built  into  LOCIS  in  the  schedule  screen,  the  data  was  not  available  in  the  prototype  used 
for  this  study.  It  is  important  to  note,  however,  that  participants  indicated  that  they  would 
like  to  have  the  flying  schedule  for  tomorrow  available  in  the  status  at  a  glance  format  in 
addition  to  the  schedule  format.  It  may  be  fruitful  to  investigate  the  feasibility  of 
incorporating  future  flying  schedules  into  the  status  at  a  glance  format. 
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Finally,  participants  noted  that  they  could  not  obtain  information  about  shipping  status, 
the  reason  for  a  deviation,  or  landing  discrepancies  on  LOCIS.  Each  of  these  is  important 
to  the  maintenance  supervisor  in  building  SA.  However,  it  may  not  be  feasible  to  include 
this  type  of  data  in  LOCIS.  It  may  make  sense  for  maintenance  supervisors  to  continue 
to  rely  on  direct  access  to  existing  data  sources  (i.e.,  MASS,  CFRS)  for  this  type  of 
information. 

4.4.2  Scenario  2:  deployment  planning 
4.4.2 . 1  S A  findings 

Participants  were  able  to  find  the  needed  information  to  build  accurate  SA  in  the 
deployment  planning  task  presented  in  scenario  2. 


Currently,  in  building  SA  during  deployment  planning,  maintenance  supervisors  review  a 
matrix,  termed  a  rack  and  stack,  created  by  Plans  and  Schedules.  The  rack  and  stack 
displays  jets  selected  for  the  deployment  in  one  column.  Information  about  scheduled 
activities  and  status  for  each  aircraft  fill  out  the  rest  of  the  columns  in  the  matrix.  The 
rack  and  stack  is  the  primary  tool  used  over  time  to  plan  and  re-plan  a  deployment. 

In  LOCIS,  the  user  enters  specifications  for  a  planned  deployment  in  the  health  of  the 
fleet  forecasting  tool,  and  LOCIS  generates  a  rack  and  stack. 

Using  the  LOCIS  health  of  the  fleet  forecasting  tool,  all  of  the  participants  noted  that 
LOCIS  recommended  a  number  of  aircraft  that  were  scheduled  for  PDM  (major 
scheduled  maintenance  off-site).  Each  participant  explained  that  a  scheduled  PDM 
would  eliminate  that  jet  from  the  planned  deployment  and  they  quickly  and  easily 
removed  that  jet  from  the  rack  and  stack.  Although  this  suggests  that  the  LOCIS 
recommendations  were  not  accepted  carte  blanche  by  participants,  it  also  indicates  that 
participants  were  engaged  in  the  scenario  and  had  sufficient  SA  to  quickly  notice  flaws  in 
the  recommended  plan  and  rapidly  make  the  necessary  adjustments. 

Other  important  data  that  was  noted  by  all  participants  include  HPO,  which  is  another 
strong  determinant  as  to  whether  a  specific  aircraft  will  be  available  for  a  deployment. 

All  agreed  that  an  aircraft  generally  cannot  undergo  phase  inspection  while  deployed,  but 
that  some  schedule  and  flying  manipulations  could  be  made  so  that  the  aircraft  could 
have  the  inspection  before  or  after  the  deployment. 

Participants  noticed  TCTOs  (time  sensitive  system  upgrades)  scheduled.  They  also 
identified  inspections  that  require  grounding  such  as  18-month  gun  inspections  and 
weight  and  balance  inspections.  Inspections  such  as  egress  final  and  parachute  are  more 
flexible  and  do  not  result  in  grounding. 

In  addition  to  noticing  most  of  the  same  things  and  interpreting  the  information  in  a 
consistent  manner,  participants  forecasted  the  situation  similarly.  All  indicated  that 
situation  seemed  challenging  but  that  they  were  highly  confident  that  they  would  be  able 
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to  make  the  deployment.  This  is  typical  of  maintenance  supervisor’s  attitudes,  they 
harness  resources  and  make  adjustments  to  meet  the  mission  need  -  no  matter  how 
challenging  the  tasks  involved  with  meeting  the  goal.  In  this  context,  participants  had  the 
awareness  to  note  that  the  scenario’s  PDM  schedule  did  not  accurately  reflect  PDM 
schedules  in  the  real  world.  They  were  confident  that  over  the  course  of  four  weeks,  they 
would  be  able  to  work  out  any  problems  with  the  schedules  to  meet  the  deployment 
needs. 

4.4.2.2  LOCIS  findings 

The  health  of  the  fleet  forecasting  tool  in  LOCIS  supported  S  A  for  the  maintenance 
supervisor  working  on  the  deployment  scenario.  The  calendar  format  allowed  the 
participants  to  scan  the  computer  generated  recommendations,  as  well  as  key  information 
associated  with  each  jet.  Recommendations  are  presented  in  a  prioritized  list,  so 
participants  were  able  to  see  which  aircraft  were  rated  highest  by  the  forecasting  tool  as 
well  as  those  that  received  lower  ratings.  Furthermore,  the  presentation  format  allowed 
the  participants  to  assess  each  aircraft’s  suitability  without  relying  completely  on  the 
computer  recommendations.  Information  required  to  assess  each  jet’s  suitability  for  the 
deployment  includes  Program  Depot  Maintenance  (PDM)  status  (a  scheduled  activity  that 
cannot  be  changed);  flying  time;  inspection  status,  including  FSL  and  In  Flight  Refueling 
(IFR);  Time  Critical  Technical  Order  (TCTO)  status;  and  schedule  of  HPOs  (Hourly  Post 
Operations)  and  dates  due. 

We  observed  participants  examine  the  recommendations  generated  by  LOCIS,  and  tweak 
the  rack  and  stack  based  on  experience  and  individual  judgment.  Observation  and 
interview  data  indicate  that  users  found  the  LOCIS  tool  to  be  valuable,  the  automatically 
generated  rack  and  stack  to  be  a  logical  starting  place,  and  that  the  data  displayed 
provided  maintenance  supervisors  the  information  needed  to  make  important  judgments 
based  on  their  own  experience. 

We  anticipate  that  LOCIS  will  represent  considerable  time  savings  for  deployment 
planning,  while  maintaining  a  high  level  of  S  A  for  the  maintenance  supervisor. 
Participants  reported  that  it  currently  takes  a  half-day  for  Plans  and  Schedules  to  gather 
information  from  the  CAMS  terminal,  and  a  day  to  manually  compile  the  information  for 
the  rack  and  stack.  It  is  expected  that,  using  LOCIS,  the  maintenance  supervisor  will  be 
able  to  complete  this  task  in  less  than  an  hour  and  then  will  be  able  to  quickly  and  easily 
revise  the  rack  and  stack  throughout  the  period  leading  up  to  the  deployment.  This 
capability  is  currently  unavailable  to  the  maintenance  supervisor  without  LOCIS. 

In  addition,  some  information  that  was  previously  available  only  in  anecdotal  form,  is 
displayed  in  LOCIS  using  historical  data  to  provide  a  more  accurate  picture.  For 
example,  an  aircraft  may  have  a  reputation  of  being  a  “hangar  queen”  because  it  is  prone 
to  breaking.  Maintenance  supervisors  take  trends  in  aircraft  breakage  into  account  when 
planning  for  a  deployment,  but  currently  have  to  rely  on  assimilating  data  from  various 
sources  and  on  memory  and  judgment  for  this  type  of  information.  LOCIS  provides 
systems  integrity  data  (Low  Integrity)  that  is  based  on  assimilating  the  actual  breakage 
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history  for  each  specific  aircraft.  This  capability  was  perceived  by  participants  as 
extremely  valuable. 

One  participant  reported  that  having  Hourly  Post  Operations  (HPO)  Difference  data 
available  numerically  in  LOCIS,  rather  than  graphically  as  it  is  currently  provided  would 
help  him  in  making  confident  forecasts  during  planning.  HPO  Difference  data  is  one 
measure  of  how  many  hours  an  aircraft  can  fly  before  it  must  go  into  a  phased  inspection, 

and  therefore  has  significant  impact  on  deployment  planning. 


Participants  noted  several  areas  in  which  the  health  of  the  fleet  forecasting  tool  could  be 
improved.  Specifically,  Engine  Time  changes  are  not  displayed  on  the  rack  and  stack 
screen;  yet  this  is  important  information  used  in  deployment  planning  for  many  aircraft. 
One  participant  noted  that  there  are  some  aircraft-specific  inspections  that  he  would  like 
to  have  available  on  the  rack  and  stack  screen.  Another  participant  comment  referred  to 
the  length  of  time  to  complete  certain  types  of  inspections  as  valuable  information  in 
deployment  planning. 


One  participant  suggested  that  LOCK  should  identify  individual  aircraft  as  potential 
deployers  on  status  display  screens  (e.g.,  the  at-a-glance  screens),  so  there  is  shared 
understanding  of  which  aircraft  are  targeted  for  the  planned  deployment  activity.  Others 
commented  on  the  need  for  information  regarding  what  parts  have  been  loaded  on  pallets 
and  which  parts  are  available  for  palleting.  There  was  also  a  suggestion  that  LOCK 

should  provide  more  information  regarding  which  systems  are  keeping  the  Low  Integrity 
scores  down.  J 

4.4.3  Scenario  3:  shift  change  recap 

4.4.3. 1  SA  findings 

Due  to  technically  difficulties,  data  regarding  the  flying  schedule  for  the  following  day’s 
flying  schedule  was  not  available  in  the  LOCK  prototype  used  in  this  study.  As  a  result, 
participants  were  not  able  to  build  adequate  SA  for  the  shift  change  recap  scenario. 
Typically,  the  maintenance  supervisor  examines  the  following  day’s  flying  schedule 
along  with  the  current  status  of  the  jets  in  the  unit  in  order  to  recap  the  events  of  the  day 
and  highlight  priorities  for  the  maintenance  supervisor  on  the  next  shift.  The  limitations 
in  the  data  contained  in  the  prototype  used  in  this  study  made  this  impossible. 

4.4.3.2  LOCK  findings 

In  spite  of  the  difficulties  in  assessing  the  impact  of  LOCK  on  the  maintenance 
supervisor’s  SA  in  the  shift  change  recap  task,  we  were  able  to  elicit  from  participants 
about  the  types  of  information  they  would  like  to  have  that  are  not  currently  available  in 


In  order  to  support  SA  for  the  end-of-day  recap,  LOCK  must  provide  future  flying 
schedule’s  (i.e.,  the  schedule  for  the  week)  in  addition  to  current  status.  Specifically, 
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participants  reported  a  need  for  real-time  information  in  the  recap  screen.  Knowledge  of 
the  status  of  each  jet  after  each  sortie  is  key  to  having  up-to-the  minute  SA.  In  addition, 
as  mentioned  in  the  context  of  scenario  2,  it  may  be  useful  to  depict  tomorrow’s  flying 
schedule  in  the  status  at  a  glance  format,  in  addition  to  the  schedule  screen. 

It  would  also  be  useful  to  provide  more  information  about  scheduled  maintenance  and  the 
current  status  of  maintenance.  Participants  referred  to  the  “checkerboard”  format  they 
currently  use  as  an  invaluable  tool.  It  may  be  possible  for  future  versions  of  LOCIS  to 
pull  data  from  the  CAMS  380  screen  to  create  a  checkerboard  in  a  format  familiar  to 
maintenance  supervisors,  depicting  more  detailed  information  about  the  scheduled 
maintenance.  Some  participants  also  suggested  that  highlighting  the  jet  that  has  been 
cannibalized  to  repair  others  (i.e.,  the  CANN  jet)  would  be  helpful. 

5  Lessons  Learned  about  Measuring  SA 

This  initial  attempt  to  assess  SA  in  a  command  and  control  task  has  highlighted  many 
important  issues  to  be  considered  in  future,  larger-scale  studies.  As  this  preliminary 
study  was  conducted  in  preparation  for  a  larger  study  to  assess  the  impact  of  a  fielded 
version  of  LOCIS  at  a  working  AMU,  lessons  learned  will  be  discussed  in  that  context.  It 
will  be  important  to  frame  the  full-scale  SA  field  evaluation  of  LOCIS,  realistically 
considering  the  type  of  data  that  it  is  possible  to  collect  and  the  context  in  which  it  should 
be  interpreted.  Planning  the  field  study  will  require  a  great  deal  of  upfront  preparation  to 
ensure  that  scenarios  presented  to  the  maintenance  supervisor  are  robust  enough  to  gather 
meaningful  SA  data.  This  is  true  whether  the  scenario  is  simulated  or  actual. 

Additionally,  the  evaluation  must  use  methods  that  provide  meaningful,  verifiable  SA 
data  elements.  The  lessons  learned,  as  highlighted  in  the  paragraphs  below,  are  intended 
to  provide  insight  as  to  how  this  type  of  investigation  might  be  successfully  carried  out  in 
a  fixll-scale  SA  field  evaluation  of  LOCIS. 

5.1  Framing  a  field  study 

Maintenance  supervisors  are  highly  efficient  logisticians  working  in  a  complex, 
demanding  environment.  The  risks  associated  with  unsuccessful  performance  are 
potentially  life  threatening;  therefore,  motivation  is  very  high.  Maintenance  supervisors 
have  developed  tools  and  techniques  necessary  to  provide  them  with  superior  S  A.  The 
problem  with  these  tools  and  techniques  is  that  they  are  very  time  consuming  and  they 
vary  from  one  base  to  another.  If  maintenance  supervisors  deploy  to  a  different  site,  they 
need  to  learn  new  tools  and  techniques  to  successfully  perform  their  jobs.  Standardized 
tools  and  techniques  are  needed,  and  LOCIS  provides  this  opportunity  for 
standardization.  Given  this  frame  of  reference,  the  success  of  LOCIS  in  an  SA  evaluation 
should  focus  on  assurance  of  maintained  good  S  A,  not  on  improving  S  A.  That  is, 
maintenance  supervisors  already  have  superior  S  A;  therefore,  a  standardized  tool  like 
LOCIS  should  assist  in  maintaining  that  excellent  SA  while  significantly  reducing 
information  gathering  time  associated  with  current  SA  tools  and  techniques. 
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In  addition,  the  full-scale  field  study  will  allow  future  data  collection  to  incorporate  the 
concept  of  team  S  A  as  a  means  to  more  effectively  measure  SA  in  the  maintenance 
supervisor  task.  Data  collection  methods  should  take  advantage  of  team  SA  to  determine 
common  cues,  understanding  of  the  cues,  and  forecasting  of  future  events.  For  example, 
shift  changes  between  first  and  second  shift  maintenance  supervisors  could  provide 
extremely  useful  opportunities  to  assess  the  value  of  LOCIS  in  terms  of  providing  all  the 
necessary  SA  elements  for  successful  job  performance.  This  type  of  evaluation  would 
compare  2nd  shift  maintenance  supervisor  SA  when  using  LOCIS  and  when  using 
common  tools  and  techniques  for  information  gathering.  Again,  the  focus  would  be  on 
assuring  that  LOCIS  provides  no  reduction  in  SA,  but  does  provide  significant  savings  in 
terms  of  time  spent  collecting,  integrating  and  disseminating  information. 

5.2  Scenario  selection/creation 

The  primary  advantage  of  simulated  scenarios  is  that  they  allow  the  investigators  to 
control  aspects  of  the  study.  The  same  scenario  can  be  presented  to  multiple  subject 
matter  experts  (SMEs)  for  comparison.  It  is  possible  to  build  in  specific,  difficult 
elements  to  evaluate  the  system’s  ability  to  support  users  in  challenging  contexts. 
Investigators  can  specify  a  time-frame  for  the  data  collection  and  ask  SMEs  to  schedule 
time  for  participation. 

However,  if  pre-built  scenarios  are  used  for  the  field  study,  it  will  be  important  to  ensure 
that  dependencies  among  planned  tasks  are  robust.  Scenarios  must  reflect  the 
dependencies  among  planned  activities.  In  the  preliminary  study,  the  three  scenarios 
used  were  relatively  independent,  which  in  some  ways  simplified  the  maintenance  task 
and  made  for  a  cleaner  analysis.  We  believe  this  artificiality  limited  our  ability  to 
realistically  assess  important  components  of  SA  for  the  maintenance  supervisor.  Because 
of  this,  we  would  recommend  using  more  complex  scenarios  with  greater 
interdependencies  in  future  studies. 

Similarly,  if  actual  operating  environments  are  used,  it  will  be  important  to  ensure  that 
dependencies  among  planned  tasks  are  emphasized  in  data  collection.  The  primary 
advantage  of  using  the  actual  operating  environment  is  that  the  real-world  setting  adds  a 
level  of  credibility  to  findings  that  cannot  be  attained  with  simulated  scenarios. 

Collecting  data  as  maintenance  supervisors  go  about  their  jobs  eliminates  the  difficulty 
involved  in  creating  complex,  realistic  scenarios  complete  with  interdependencies  and  the 
important  artifacts  associated  with  the  maintenance  supervisor’s  job  (i.e.,  email,  phone 
calls,  radio  traffic).  However,  collecting  data  in  the  operational  environment  is  not  a 
trivial  endeavor.  Investigators  must  be  careful  not  to  interfere  with  maintenance 
operations,  and  be  willing  to  abandon  data  collection  if  unexpected  events  occur  that 
require  participants’  full  attention.  Furthermore,  it  is  not  possible  to  schedule 
challenging,  interdependent  tasks  in  the  real  world.  Investigators  may  be  required  to 
observe  operations  for  a  period  of  time  before  appropriate  scenarios  that  challenge  SA 
arise. 
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If  possible,  we  recommend  the  use  of  a  combination  of  simulated  and  real-world 
scenarios  for  the  full-scale  LOCIS  field  study.  Regardless  of  whether  a  combination  is 
used,  or  one  approach  is  chosen  over  the  other  due  to  practical  constraints,  scenarios  — 
whether  pre-built  or  actual  --  need  to  include  interdependent  complexities  to  acquire  full 
comprehension  of  the  SA  elements  being  considered  by  the  maintenance  supervisor. 

5.3  Tailoring  SA  measurement  tools 

We  found  this  approach  of  measuring,  whether  participants  in  our  study  were 

A)  noticing  the  same  cues, 

B)  understanding  the  cues  in  the  same  way,  and 

C)  forecasting  events  in  the  future  in  a  similar  manner, 

to  be  effective.  Of  course,  finding  effective  means  to  uncover  what  participants  are 
noticing,  understanding,  and  forecasting  as  they  perform  various  tasks  is  the  most 
challenging  part  of  measuring  SA.  We  found  participants  to  be  surprisingly  comfortable 
with  the  think  aloud  during  the  simulation  scenarios  used  in  this  preliminary  study. 
Explaining  what  information  they  were  noticing,  what  information  they  were  seeking, 
and  what  meaning  it  had  for  them  did  not  seem  to  represent  a  significant  distraction  from 
the  task.  Using  the  think  aloud  in  combination  with  direct  follow-up  questions  allowed 
participants  to  elaborate  on  things  noted  during  the  task  immediately  following 
completion.  We  would  recommend  this  combination  of  data  collection  methods  for 
future  studies  in  which  simulated  scenarios  are  used. 

However,  if  data  is  collected  in  the  operational  environment,  think  aloud  techniques  may 
be  too  intrusive.  In  a  real-world  setting,  investigators  will  likely  rely  on  observation  of 
the  maintenance  supervisor,  combined  with  follow-up  interviews.  Although  this 
represents  a  greater  reliance  on  the  participants’  memory  of  what  was  noticed  and 
understood  (and  therefore  an  increased  likelihood  of  memory  error),  interviews 
conducted  immediately  following  the  observation  sessions  will  minimize  memory 
degradation. 

In  terms  of  data  analysis,  the  preliminary  study  strategy  of  using  qualitative  analysis 
methods  to  assess  whether  participants  notice  and  understand  the  significance  of  specific 
elements  built  into  a  simulated  scenario  proved  useful.  We  would  recommend  this 
approach  in  the  future,  if  simulated  scenarios  are  used. 

However,  an  even  stronger  argument  for  the  effectiveness  of  LOCIS  in  supporting  SA 
could  be  made  if  data  from  the  operational  environment  could  be  used  to  compare  the  S  A 
of  maintenance  supervisors  using  LOCIS  with  that  of  maintenance  supervisors  relying  on 
traditional  means  of  building  SA.  Similar  analysis  methods  would  be  used.  This  type  of 
direct  comparison  will  likely  be  difficult  to  arrange,  but  should  be  a  goal  as  the  study  plan 
is  created. 
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Finally,  we  recommend  investigating  the  use  of  team  S  A  as  a  means  to  assess  SA  in  the 
field  study.  As  described  earlier  in  this  paper,  we  would  expect  maintenance  supervisors 
working  different  shifts  to  have  very  similar  SA.  Given  the  difficulty  in  establishing  a 
ground  truth  SA  with  which  to  compare  participant  responses,  comparing  responses  from 
team  members  expected  to  have  shared  SA  may  prove  to  be  a  valuable  measure. 

6  Conclusions 


6.1  Measuring  SA 

The  job  of  the  maintenance  supervisor  has  more  in  common  with  tasks  such  as  command 
and  control  that  involve  monitoring  over  a  longer  timeline,  than  with  piloting.  The 
concept  of  situation  awareness  is  instantiated  quite  differently  in  these  domains  and 
therefore  requires  different  strategies  for  measurement.  This  preliminary  test  represents 
an  initial  step  toward  tailoring  existing  SA  measurement  techniques  for  use  in  command 
and  control  tasks. 

In  this  preliminary  study,  we  were  successfully  able  to  collect  SA  data  from  experienced 
maintenance  supervisors  interacting  with  LOCIS  in  the  context  of  simulated  scenarios. 
Data  analysis  revealed  that,  in  two  of  the  three  scenarios,  participants  independently 
noticed  the  same  key  elements,  developed  a  common  understanding  of  the  situation,  and 
generated  similar  forecasts.  Technical  difficulties  precluded  the  collection  of  this  type  of 
data  for  the  third  scenario. 

The  preliminary  study  highlighted  important  issues  to  be  considered  in  the  design  of  the 
LOCIS  field  study.  These  include: 

•  The  maintenance  community  has  evolved  a  set  of  processes  and  tools  that  allow 
maintenance  supervisors  to  quickly  obtain  up-to-the-minute,  superior  SA.  The 
LOCIS  field  study  should  aim  to  show  that  LOCIS  provides  added  efficiency, 
without  reducing  SA.  It  is  unrealistic  to  expect  that  the  use  of  LOCIS  will 
increase  SA. 

•  If  possible,  a  combination  of  data  from  simulated  scenarios  and  data  from  actual 
operating  environments  should  be  collected. 

•  Regardless  of  whether  simulated  scenario,  actual  operation  environments,  or  a 
combination  of  the  two  are  used,  it  will  be  important  to  ensure  that  dependencies 
among  planned  tasks  are  emphasized. 

•  The  combination  of  the  think  aloud  techniques  and  direct  questions  followed  by 
qualitative  data  analysis  should  continue  to  be  refined  for  file  field  study. 

•  The  full-scale  field  study  should  take  advantage  of  aspects  of  team  S  A  as  an 
additional  means  to  assess  the  impact  of  LOCIS  on  SA. 

6.2  Recommendations  for  LOCIS 

Data  from  this  small,  preliminary  study  indicate  that  LOCIS  successfully  supports  the 
development  of  accurate  SA  in  two  tasks  key  to  the  maintenance  supervisor  job. 
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Specifically,  in  preparing  for  the  Daily  Aircraft  Maintenance  Morning  meeting, 
participants  were  able  to  quickly  obtain  a  high  level  overview  of  the  status  of  the  unit  and 
gather  detailed  data  about  the  status  of  specific  jets,  as  needed.  In  the  context  of 
preparing  for  a  deployment,  participants  were  able  to  use  the  health  of  the  fleet 
forecasting  tool  to  obtain  information  about  the  status  and  planned  activities  for  each  jet 
in  the  unit.  Furthermore,  they  were  able  to  use  this  tool  to  consider  different  options,  and 
the  implications  of  various  courses  of  action.  These  sorts  of  cognitive  activities  are  key 
to  building  and  maintaining  accurate  SA.  Although  our  initial  data  plan  included  an 
assessment  of  the  impact  of  LOCIS  on  the  maintenance  supervisor’s  ability  to  build 
accurate  SA  in  the  context  of  an  end-of-day  recap  at  shift  change,  we  were  not  able  to 
collect  this  data  due  to  technical  difficulties. 

This  preliminary  study  also  revealed  a  set  of  suggested  improvements  to  LOCIS.  These 
include: 

•  Tomorrow’s  flying  schedule.  Participants  report  that  they  are  always  thinking  at 
least  a  day  ahead.  Displaying  the  following  day’s  flying  schedule  in  the  status  at 
a  glance  format  would  be  very  helpful.  A  second  solution  suggested  the  addition 
of  an  identifier  to  the  current  status  at  a  glance  screen  for  jets  that  are  scheduled 
to  fly  the  following  day. 

•  Aircraft  summary  screen.  Participants  emphasized  the  fact  that  much  of  the  day 
they  are  away  from  their  desks  and  their  computers.  They  would  like  to  have  a 
printout  that  details  all  the  broken  aircraft,  the  driving  discrepancy  for  each,  and 
the  corresponding  ETIC. 

•  Real-time  recap.  It  is  not  enough  to  provide  recap  information  for  the  previous 
day.  Participants  indicated  that  they  need  to  know  after  each  flight:  landing  codes 
for  each  jet,  deviations,  plus  the  reason  for  each  deviation. 

•  Scheduled  maintenance.  The  existing  scheduled  maintenance  information  could 
be  improved  to  support  SA.  Users  would  like  to  see  a  screen  formatted  in  manner 
similar  to  the  “checkerboard”  they  currently  use.  This  screen  would  display  the 
maintenance  that  has  been  scheduled  for  the  entire  week. 

The  preliminary  study  summarized  in  this  report  represents  an  important  step  forward  in 
assessing  SA  in  command  and  control  tasks.  The  preliminary  study  also  elicited 
important  information  regarding  the  value  of  LOCIS  in  supporting  SA  for  maintenance 
supervisors,  and  suggestions  for  improving  LOCIS.  We  anticipate  that  the  full-scale  field 
study  will  allow  us  to  continue  to  refine  our  methods,  and  that  it  will  provide  a  solid 
understanding  of  the  impact  LOCIS  has  on  the  maintenance  supervisors  ability  to  build 
and  maintain  accurate  S  A. 
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